Initial deployment: UHC Medical Policy Chatbot

RAG chatbot for UnitedHealthcare medical policies using:
- MedEmbed (1024-dim) for semantic retrieval
- Qdrant Cloud for vector search
- Groq API (Llama 3.1 8B) for answer generation
- Streamlit web UI with streaming and source citations

Made-with: Cursor

.env.example

@@ -0,0 +1,8 @@
+# Qdrant - local docker (default)
+QDRANT_HOST=localhost
+QDRANT_PORT=6333
+QDRANT_COLLECTION=uhc_policies
+
+# Qdrant Cloud (uncomment to use instead of local)
+# QDRANT_URL=https://your-cluster.qdrant.io
+# QDRANT_API_KEY=your-api-key-here

README.md

@@ -1,12 +1,232 @@
 ---
-title: Uhc Policy Chatbot
-emoji: 🏆
-colorFrom: green
+title: UHC Medical Policy Chatbot
+emoji: 🏥
+colorFrom: blue
 colorTo: purple
-sdk: gradio
-sdk_version: 6.9.0
+sdk: streamlit
+sdk_version: 1.44.1
 app_file: app.py
 pinned: false
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# UHC Medical Policy Chatbot
+
+A RAG-powered chatbot that answers questions about UnitedHealthcare (UHC) medical policies. Built for doctors, hospital staff, and insurance coordinators who need accurate, cited answers about coverage criteria, CPT/HCPCS codes, and medical necessity requirements.
+
+## Hosted Chatbot
+
+**URL:** [https://huggingface.co/spaces/mxp1404/uhc-policy-chatbot](https://huggingface.co/spaces/mxp1404/uhc-policy-chatbot)
+
+### How to Use — Step-by-Step
+
+1. Open the link above in your browser.
+2. Wait for the model to load (first visit takes ~30 seconds for MedEmbed to initialize).
+3. Type your question in the chat input at the bottom — for example:
+   - *"Is bariatric surgery covered for BMI over 40?"*
+   - *"What documentation is needed for gender-affirming surgery?"*
+   - *"Are intrapulmonary percussive ventilation devices covered for home use?"*
+4. The chatbot will search relevant policy chunks, then stream an answer with citations.
+5. Click **"📚 Sources"** below each answer to see the exact policy sections used.
+6. Use **"🗑️ Clear conversation"** in the sidebar to start a new session.
+
+The chatbot only answers from official UHC policy documents — it will tell you if it doesn't have enough information rather than guessing.
+
+---
+
+## Architecture
+
+### High-Level Design (HLD)
+
+```
+┌─────────────┐     ┌──────────────────────────────────────────────┐
+│   Browser    │────▶│  Streamlit App (HuggingFace Spaces)          │
+│   (User)     │◀────│                                              │
+└─────────────┘     │  ┌─────────────┐    ┌─────────────────────┐  │
+                    │  │ MedEmbed    │    │ Groq API            │  │
+                    │  │ (1024-dim)  │    │ Llama 3.1 8B        │  │
+                    │  │ cached RAM  │    │ 560 tok/s           │  │
+                    │  └──────┬──────┘    └──────▲──────────────┘  │
+                    │         │                   │                  │
+                    │         ▼                   │                  │
+                    │  ┌─────────────┐   context + query            │
+                    │  │ Qdrant Cloud│────────────┘                 │
+                    │  │ (vectors)   │                              │
+                    │  └─────────────┘                              │
+                    └──────────────────────────────────────────────┘
+```
+
+**Data flow for each query:**
+
+1. User types a question in the Streamlit chat interface
+2. The query is encoded into a 1024-dimensional vector using **MedEmbed** (loaded once, cached in memory)
+3. The vector is sent to **Qdrant Cloud** for similarity search — returns top-K policy chunks with metadata
+4. Retrieved chunks are deduplicated, truncated, and formatted into a context block
+5. The context + query + system prompt are sent to **Groq API** (Llama 3.1 8B) for answer generation
+6. The response is streamed token-by-token back to the user with source citations
+
+### Low-Level Design (LLD)
+
+#### Project Structure
+
+```
+uhc/
+├── app.py                          # Streamlit web UI entry point
+├── requirements.txt                # Python dependencies
+├── .env.example                    # Environment variable template
+│
+├── chatbot/                        # Chatbot application layer
+│   ├── config.py                   # Centralized config (LLM, retrieval, env vars)
+│   ├── retriever.py                # PolicyRetriever: MedEmbed + Qdrant wrapper
+│   ├── llm_groq.py                 # Groq API client (deployed)
+│   ├── llm.py                      # Ollama client (local dev)
+│   ├── prompts.py                  # System prompt, context formatting, deduplication
+│   └── cli.py                      # CLI interface (local dev)
+│
+├── embedding/                      # Embedding pipeline
+│   └── scripts/
+│       ├── config.py               # Embedding model + Qdrant connection config
+│       ├── embed_chunks.py         # Generate embeddings from RAG chunks
+│       ├── store_qdrant.py         # Upsert embeddings into Qdrant with payload indexes
+│       ├── search.py               # Standalone search CLI for testing
+│       └── test_retrieval.py       # Batch retrieval evaluation (10 test cases)
+│
+└── scraper/                        # Data ingestion pipeline
+    ├── download_policies.py        # Scrape PDFs from UHC website
+    ├── extract_pdf_text.py         # PDF → structured sections with metadata
+    ├── create_rag_chunks.py        # Section-aware semantic chunking
+    └── data/processed/
+        ├── extracted_sections.json # Extracted text per policy/section
+        └── rag_chunks.json         # Final RAG chunks with metadata
+```
+
+#### Module Design
+
+**`chatbot/retriever.py` — PolicyRetriever**
+- Loads `abhinand/MedEmbed-large-v0.1` (1024-dim medical embeddings) via `sentence-transformers`
+- Connects to Qdrant Cloud; supports both cloud and local Qdrant
+- Encodes queries → cosine similarity search → returns `ChunkResult` dataclasses
+- Filters out low-value sections (References, Application) that pollute results
+- Boosts Coverage Rationale chunks (+0.02 score) so authoritative coverage statements always surface
+- Retry logic with exponential backoff for transient Qdrant errors
+
+**`chatbot/prompts.py` — Prompt Engineering**
+- System prompt enforces: answer from context only, 2–4 bullet points, cite sources, coverage-awareness
+- `deduplicate_chunks()` keeps highest-scoring chunk per (policy, section) pair
+- `format_context()` truncates each chunk to 800 chars at sentence boundaries, caps total at 6000 chars
+- Coverage Rationale is explicitly marked as authoritative for coverage decisions
+
+**`chatbot/llm_groq.py` — GroqClient**
+- Uses `groq` Python SDK with streaming chat completions
+- Graceful rate-limit handling (Groq free tier: 250K TPM)
+- Same `chat_stream()` / `chat()` interface as the Ollama client for interchangeability
+
+**`scraper/extract_pdf_text.py` — PDF Extraction**
+- Paragraph-level extraction using `pdfplumber` (not line-by-line)
+- Robust header/footer/sidebar removal with regex patterns
+- Structured metadata parsing: policy number, effective date, plan type, document type
+- Table extraction support; skips boilerplate sections and HTML-disguised files
+
+**`scraper/create_rag_chunks.py` — Semantic Chunking**
+- Section-aware chunking: different strategies per section type
+  - Coverage Rationale → criteria-based splitting
+  - Applicable Codes → table-aware chunking
+  - Clinical Evidence → study-based splitting
+  - Others → paragraph-aware with sentence-boundary overlap
+- Rich metadata per chunk: policy name, section, plan type, page range, provider
+- Deterministic chunk IDs for deduplication during re-indexing
+
+**`embedding/scripts/embed_chunks.py` — Embedding Generation**
+- Prepends metadata to chunk text before encoding for better retrieval
+- Batch processing (32 chunks at a time) with GPU/MPS/CPU auto-detection
+- Saves to `.npz` for efficient storage and reloading
+
+**`embedding/scripts/store_qdrant.py` — Vector Storage**
+- Creates Qdrant collection with cosine distance
+- Upserts embeddings with full metadata payloads
+- Creates payload indexes on `section`, `policy_name`, `plan_type`, `doc_type`, `provider` for efficient filtered search
+
+#### Edge Cases Handled
+
+| Edge Case | Handling |
+|---|---|
+| Empty / whitespace query | Warning message, no API call |
+| Qdrant connection failure | Retry with exponential backoff (3 attempts) |
+| Groq rate limit (429) | Caught and shown as user-friendly message |
+| No relevant chunks found | "I don't have enough policy information" |
+| Coverage vs. evidence conflict | System prompt + Coverage Rationale boost ensures correct answer |
+| Very long conversation | History trimmed to last 3 turns |
+| Model loading on first visit | Spinner shown; cached with `st.cache_resource` |
+
+---
+
+## Extending for Other Insurance Providers
+
+The system is designed for multi-provider extensibility:
+
+1. **Data layer**: Each chunk in Qdrant has a `provider` field (currently `"UnitedHealthcare"`). Adding a new provider means running the same pipeline with a new provider slug — chunks coexist in the same collection.
+
+2. **Scraper**: `scraper/download_policies.py` can be adapted for any provider's website. The extractor and chunker handle standard medical policy PDF structures.
+
+3. **Embedding**: The same MedEmbed model works for all medical content. New provider chunks are embedded and upserted alongside existing ones.
+
+4. **Retrieval**: Add a `provider_filter` parameter to narrow results by provider, or query across all providers simultaneously.
+
+5. **UI**: Add a provider selector dropdown in the Streamlit sidebar — one line change.
+
+```python
+# Example: adding Aetna
+retriever.retrieve(query, provider_filter="aetna")
+```
+
+---
+
+## Local Development Setup
+
+```bash
+# 1. Clone the repo
+git clone https://github.com/<your-username>/uhc-policy-chatbot.git
+cd uhc-policy-chatbot
+
+# 2. Create virtual environment
+python3 -m venv venv
+source venv/bin/activate
+
+# 3. Install dependencies
+pip install -r requirements.txt
+
+# 4. Configure environment variables
+cp .env.example .env
+# Edit .env with your Qdrant and Groq API keys
+
+# 5. Run the Streamlit app
+streamlit run app.py
+
+# Or use the CLI with Ollama (local LLM)
+ollama serve &
+ollama pull phi3.5
+python -m chatbot.cli
+```
+
+### Environment Variables
+
+| Variable | Description | Required |
+|---|---|---|
+| `QDRANT_URL` | Qdrant Cloud cluster URL | Yes |
+| `QDRANT_API_KEY` | Qdrant Cloud API key | Yes |
+| `QDRANT_COLLECTION` | Collection name (default: `uhc_policies`) | No |
+| `GROQ_API_KEY` | Groq API key ([get free](https://console.groq.com/keys)) | Yes (web) |
+| `GROQ_MODEL` | Groq model (default: `llama-3.1-8b-instant`) | No |
+
+---
+
+## Tech Stack
+
+| Component | Technology |
+|---|---|
+| Embedding Model | [MedEmbed-large-v0.1](https://huggingface.co/abhinand/MedEmbed-large-v0.1) (1024-dim) |
+| Vector Database | [Qdrant Cloud](https://qdrant.tech/) |
+| LLM (deployed) | [Llama 3.1 8B](https://console.groq.com/) via Groq (560 tok/s) |
+| LLM (local dev) | Phi-3.5 Mini via Ollama |
+| Web Framework | Streamlit |
+| Hosting | HuggingFace Spaces (free tier) |
+| PDF Extraction | pdfplumber + BeautifulSoup |

app.py

@@ -0,0 +1,194 @@
+"""
+UHC Medical Policy Chatbot — Streamlit Web UI.
+
+Deployed on HuggingFace Spaces. Uses MedEmbed for retrieval from Qdrant Cloud
+and Groq (Llama 3.1 8B) for answer generation.
+"""
+
+import time
+import streamlit as st
+
+from chatbot.config import GROQ_MODEL, RETRIEVAL_TOP_K, MAX_HISTORY_TURNS
+from chatbot.retriever import PolicyRetriever
+from chatbot.llm_groq import GroqClient, GroqError
+from chatbot.prompts import format_context, build_messages, deduplicate_chunks
+
+# ---------------------------------------------------------------------------
+# Page config
+# ---------------------------------------------------------------------------
+st.set_page_config(
+    page_title="UHC Policy Chatbot",
+    page_icon="🏥",
+    layout="centered",
+)
+
+# ---------------------------------------------------------------------------
+# Cached singletons — loaded once, shared across reruns
+# ---------------------------------------------------------------------------
+
+@st.cache_resource(show_spinner=False)
+def load_retriever() -> PolicyRetriever:
+    logs: list[str] = []
+    r = PolicyRetriever()
+    r.init(status_callback=lambda msg: logs.append(msg))
+    return r
+
+
+@st.cache_resource
+def load_llm() -> GroqClient:
+    return GroqClient()
+
+
+# ---------------------------------------------------------------------------
+# Sidebar
+# ---------------------------------------------------------------------------
+with st.sidebar:
+    st.markdown("## 🏥 UHC Policy Chatbot")
+    st.caption(f"LLM: **{GROQ_MODEL}** via Groq")
+    st.caption(f"Retrieval: **MedEmbed** → Qdrant (top-{RETRIEVAL_TOP_K})")
+    st.divider()
+
+    st.markdown("### How to use")
+    st.markdown(
+        "Ask questions about UnitedHealthcare medical policies — "
+        "coverage criteria, CPT codes, medical necessity, and more."
+    )
+    st.markdown(
+        "**Examples:**\n"
+        "- Is bariatric surgery covered for BMI over 40?\n"
+        "- What documentation is needed for gender-affirming surgery?\n"
+        "- Is HFCWO covered for cystic fibrosis?\n"
+        "- What are the criteria for whole genome sequencing?"
+    )
+    st.divider()
+
+    if st.button("🗑️ Clear conversation"):
+        st.session_state.messages = []
+        st.session_state.chunks_history = []
+        st.rerun()
+
+    st.caption(
+        "Built for the CombineHealth Technical Assignment. "
+        "Answers are generated from policy documents only."
+    )
+
+# ---------------------------------------------------------------------------
+# Init session state
+# ---------------------------------------------------------------------------
+if "messages" not in st.session_state:
+    st.session_state.messages = []
+if "chunks_history" not in st.session_state:
+    st.session_state.chunks_history = []
+
+# ---------------------------------------------------------------------------
+# Load models
+# ---------------------------------------------------------------------------
+with st.spinner("Loading MedEmbed model and connecting to Qdrant..."):
+    retriever = load_retriever()
+
+try:
+    llm = load_llm()
+except GroqError as e:
+    st.error(f"LLM initialization failed: {e}")
+    st.stop()
+
+# ---------------------------------------------------------------------------
+# Header
+# ---------------------------------------------------------------------------
+st.title("🏥 UHC Medical Policy Chatbot")
+st.caption(
+    "Ask questions about UnitedHealthcare insurance policies. "
+    "Answers are grounded in official policy documents with source citations."
+)
+
+# ---------------------------------------------------------------------------
+# Render chat history
+# ---------------------------------------------------------------------------
+for i, msg in enumerate(st.session_state.messages):
+    with st.chat_message(msg["role"]):
+        st.markdown(msg["content"])
+
+        if msg["role"] == "assistant" and i // 2 < len(st.session_state.chunks_history):
+            chunks_for_msg = st.session_state.chunks_history[i // 2]
+            if chunks_for_msg:
+                with st.expander("📚 Sources", expanded=False):
+                    for c in chunks_for_msg:
+                        st.markdown(
+                            f"- **[{c.score:.2f}]** `{c.policy_name}` — "
+                            f"{c.section} *(pages {c.page_start}–{c.page_end})*"
+                        )
+
+# ---------------------------------------------------------------------------
+# Chat input
+# ---------------------------------------------------------------------------
+if query := st.chat_input("Ask about UHC medical policies..."):
+    query = query.strip()
+
+    if not query:
+        st.warning("Please enter a question.")
+        st.stop()
+
+    st.session_state.messages.append({"role": "user", "content": query})
+    with st.chat_message("user"):
+        st.markdown(query)
+
+    # -- Retrieve -------------------------------------------------------------
+    with st.chat_message("assistant"):
+        with st.spinner("Searching policies..."):
+            t0 = time.perf_counter()
+            try:
+                chunks = retriever.retrieve(query, top_k=RETRIEVAL_TOP_K)
+            except RuntimeError as e:
+                st.error(f"Retrieval error: {e}")
+                st.stop()
+            t_retrieval = time.perf_counter() - t0
+
+        if not chunks:
+            response_text = (
+                "I don't have enough policy information to answer this question. "
+                "Try rephrasing or asking about a specific UHC policy topic."
+            )
+            st.markdown(response_text)
+            st.session_state.messages.append(
+                {"role": "assistant", "content": response_text}
+            )
+            st.session_state.chunks_history.append([])
+            st.stop()
+
+        context = format_context(chunks)
+
+        history_for_llm = []
+        turns = st.session_state.messages[:-1]
+        if len(turns) > MAX_HISTORY_TURNS * 2:
+            turns = turns[-(MAX_HISTORY_TURNS * 2):]
+        for m in turns:
+            history_for_llm.append({"role": m["role"], "content": m["content"]})
+
+        messages = build_messages(query, context, history=history_for_llm)
+
+        # -- Generate -----------------------------------------------------------
+        try:
+            t1 = time.perf_counter()
+            response_text = st.write_stream(llm.chat_stream(messages))
+            t_gen = time.perf_counter() - t1
+        except GroqError as e:
+            st.error(str(e))
+            st.stop()
+
+        # -- Sources -----------------------------------------------------------
+        deduped = deduplicate_chunks(chunks)
+        with st.expander("📚 Sources", expanded=False):
+            for c in deduped:
+                st.markdown(
+                    f"- **[{c.score:.2f}]** `{c.policy_name}` — "
+                    f"{c.section} *(pages {c.page_start}–{c.page_end})*"
+                )
+
+        st.caption(
+            f"Retrieval: {t_retrieval:.1f}s · Generation: {t_gen:.1f}s"
+        )
+
+    st.session_state.messages.append(
+        {"role": "assistant", "content": response_text}
+    )
+    st.session_state.chunks_history.append(deduped)

chatbot/cli.py

@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+"""
+Interactive CLI chatbot for querying UHC medical policies.
+
+Loads the MedEmbed model once, retrieves relevant policy chunks from Qdrant,
+and generates answers via Phi-3.5 Mini served by Ollama.
+
+Usage:
+    python -m chatbot.cli
+    python -m chatbot.cli --top-k 5 --model phi3.5
+"""
+
+import argparse
+import sys
+import time
+
+from chatbot.config import (
+    OLLAMA_MODEL,
+    RETRIEVAL_TOP_K,
+    MAX_HISTORY_TURNS,
+)
+from chatbot.retriever import PolicyRetriever
+from chatbot.llm import OllamaClient, OllamaError
+from chatbot.prompts import format_context, build_messages
+
+# -- ANSI colors --------------------------------------------------------------
+DIM = "\033[2m"
+GREEN = "\033[92m"
+CYAN = "\033[96m"
+YELLOW = "\033[93m"
+RED = "\033[91m"
+BOLD = "\033[1m"
+RESET = "\033[0m"
+
+
+def print_banner():
+    print(f"""
+{BOLD}{'=' * 64}
+  UHC Medical Policy Chatbot
+  Model: Phi-3.5 Mini via Ollama  |  Retrieval: MedEmbed + Qdrant
+{'=' * 64}{RESET}
+
+{DIM}Commands:
+  /clear   — reset conversation history
+  /debug   — show retrieved chunks for the last query
+  /quit    — exit{RESET}
+""")
+
+
+def print_sources(chunks):
+    """Print a compact list of sources used."""
+    if not chunks:
+        return
+    seen = set()
+    print(f"\n{DIM}Sources:", end="")
+    for c in chunks:
+        key = f"{c.policy_name}/{c.section}"
+        if key not in seen:
+            seen.add(key)
+            print(f"\n  [{c.score:.2f}] {c.policy_name} — {c.section}", end="")
+    print(RESET)
+
+
+def print_debug(chunks):
+    """Print full debug info for retrieved chunks."""
+    print(f"\n{YELLOW}{'─' * 64}")
+    print(f"  DEBUG: {len(chunks)} chunks retrieved")
+    print(f"{'─' * 64}{RESET}")
+    for i, c in enumerate(chunks, 1):
+        print(f"\n{YELLOW}  [{i}] score={c.score:.4f}  {c.policy_name} / {c.section}{RESET}")
+        print(f"{DIM}  Plan: {c.plan_type}  Pages: {c.page_start}-{c.page_end}")
+        preview = c.text[:300].replace("\n", " ")
+        print(f"  {preview}...{RESET}")
+    print()
+
+
+def main():
+    parser = argparse.ArgumentParser(description="UHC Policy Chatbot CLI")
+    parser.add_argument("--top-k", type=int, default=RETRIEVAL_TOP_K)
+    parser.add_argument("--model", type=str, default=OLLAMA_MODEL)
+    args = parser.parse_args()
+
+    print_banner()
+
+    # -- Check Ollama ---------------------------------------------------------
+    llm = OllamaClient(model=args.model)
+    err = llm.check_ready()
+    if err:
+        print(f"{RED}ERROR: {err}{RESET}")
+        sys.exit(1)
+    print(f"{DIM}Ollama ready ({args.model}){RESET}")
+
+    # -- Init retriever -------------------------------------------------------
+    retriever = PolicyRetriever()
+    retriever.init(status_callback=lambda msg: print(f"{DIM}{msg}{RESET}"))
+    print()
+
+    # -- REPL -----------------------------------------------------------------
+    history: list[dict] = []
+    last_chunks = []
+    debug_mode = False
+
+    while True:
+        try:
+            query = input(f"{CYAN}{BOLD}> {RESET}").strip()
+        except (KeyboardInterrupt, EOFError):
+            print(f"\n{DIM}Goodbye.{RESET}")
+            break
+
+        if not query:
+            continue
+
+        # -- Commands ---------------------------------------------------------
+        if query.lower() == "/quit":
+            print(f"{DIM}Goodbye.{RESET}")
+            break
+        if query.lower() == "/clear":
+            history.clear()
+            last_chunks.clear()
+            print(f"{DIM}History cleared.{RESET}\n")
+            continue
+        if query.lower() == "/debug":
+            if last_chunks:
+                print_debug(last_chunks)
+            else:
+                print(f"{DIM}No chunks retrieved yet.{RESET}\n")
+            continue
+
+        # -- Retrieve ---------------------------------------------------------
+        t_start = time.perf_counter()
+        try:
+            chunks = retriever.retrieve(query, top_k=args.top_k)
+        except RuntimeError as e:
+            print(f"{RED}Retrieval error: {e}{RESET}\n")
+            continue
+        t_retrieval = time.perf_counter()
+
+        last_chunks = chunks
+        context = format_context(chunks)
+
+        # -- Build messages and stream ----------------------------------------
+        messages = build_messages(query, context, history=history)
+
+        print(f"\n{GREEN}", end="", flush=True)
+        full_response = []
+        token_count = 0
+        t_first_token = None
+        try:
+            for token in llm.chat_stream(messages):
+                if t_first_token is None:
+                    t_first_token = time.perf_counter()
+                print(token, end="", flush=True)
+                full_response.append(token)
+                token_count += 1
+        except OllamaError as e:
+            print(f"{RESET}\n{RED}LLM error: {e}{RESET}\n")
+            continue
+        t_done = time.perf_counter()
+
+        print(RESET)
+
+        # -- Sources ----------------------------------------------------------
+        print_sources(chunks)
+
+        # -- Latency ----------------------------------------------------------
+        retrieval_ms = (t_retrieval - t_start) * 1000
+        first_tok_ms = ((t_first_token or t_done) - t_retrieval) * 1000
+        gen_ms = (t_done - (t_first_token or t_retrieval)) * 1000
+        total_ms = (t_done - t_start) * 1000
+        tok_per_s = token_count / (gen_ms / 1000) if gen_ms > 0 else 0
+
+        print(f"\n{DIM}{'─' * 48}")
+        print(f"  Retrieval:    {retrieval_ms:7.0f} ms")
+        print(f"  First token:  {first_tok_ms:7.0f} ms")
+        print(f"  Generation:   {gen_ms:7.0f} ms  ({token_count} tok, {tok_per_s:.1f} tok/s)")
+        print(f"  Total:        {total_ms:7.0f} ms")
+        print(f"{'─' * 48}{RESET}")
+        print()
+
+        # -- Update history ---------------------------------------------------
+        history.append({"role": "user", "content": query})
+        history.append({"role": "assistant", "content": "".join(full_response)})
+
+        if len(history) > MAX_HISTORY_TURNS * 2:
+            history = history[-(MAX_HISTORY_TURNS * 2):]
+
+
+if __name__ == "__main__":
+    main()

chatbot/config.py

@@ -0,0 +1,31 @@
+import os
+from pathlib import Path
+from dotenv import load_dotenv
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+load_dotenv(PROJECT_ROOT / ".env")
+
+# --- Ollama LLM (local dev) ---
+OLLAMA_BASE_URL = os.getenv("OLLAMA_BASE_URL", "http://localhost:11434")
+OLLAMA_MODEL = os.getenv("OLLAMA_MODEL", "phi3.5")
+
+# --- Groq LLM (deployed / default) ---
+GROQ_API_KEY = os.getenv("GROQ_API_KEY", "")
+GROQ_MODEL = os.getenv("GROQ_MODEL", "llama-3.1-8b-instant")
+
+# --- LLM shared params ---
+LLM_TEMPERATURE = float(os.getenv("LLM_TEMPERATURE", "0.2"))
+LLM_MAX_TOKENS = int(os.getenv("LLM_MAX_TOKENS", "400"))
+LLM_TOP_P = float(os.getenv("LLM_TOP_P", "0.9"))
+
+# --- Retrieval ---
+RETRIEVAL_TOP_K = int(os.getenv("RETRIEVAL_TOP_K", "6"))
+MAX_CONTEXT_CHARS = int(os.getenv("MAX_CONTEXT_CHARS", "6000"))
+MAX_CHUNK_CHARS = int(os.getenv("MAX_CHUNK_CHARS", "800"))
+EXCLUDED_SECTIONS = {"References", "Application"}
+
+# --- Conversation ---
+MAX_HISTORY_TURNS = 3
+
+# --- Embedding (re-export from embedding pipeline) ---
+EMBEDDING_SCRIPTS_DIR = PROJECT_ROOT / "embedding" / "scripts"

chatbot/llm.py

@@ -0,0 +1,129 @@
+"""
+Ollama HTTP client for Phi-3.5 Mini.
+
+Streams responses token-by-token and provides health/model availability checks.
+Uses raw requests — no extra pip packages needed beyond `requests`.
+"""
+
+import json
+from typing import Generator
+
+import requests
+
+from chatbot.config import (
+    OLLAMA_BASE_URL,
+    OLLAMA_MODEL,
+    LLM_TEMPERATURE,
+    LLM_MAX_TOKENS,
+    LLM_TOP_P,
+)
+
+
+class OllamaError(Exception):
+    pass
+
+
+class OllamaClient:
+    def __init__(
+        self,
+        base_url: str = OLLAMA_BASE_URL,
+        model: str = OLLAMA_MODEL,
+    ):
+        self.base_url = base_url.rstrip("/")
+        self.model = model
+
+    # -- Health checks --------------------------------------------------------
+
+    def is_running(self) -> bool:
+        try:
+            r = requests.get(f"{self.base_url}/api/tags", timeout=5)
+            return r.status_code == 200
+        except requests.ConnectionError:
+            return False
+
+    def is_model_available(self) -> bool:
+        try:
+            r = requests.get(f"{self.base_url}/api/tags", timeout=5)
+            if r.status_code != 200:
+                return False
+            models = r.json().get("models", [])
+            return any(
+                m.get("name", "").startswith(self.model)
+                for m in models
+            )
+        except (requests.ConnectionError, ValueError):
+            return False
+
+    def check_ready(self) -> str | None:
+        """Return an error message if not ready, else None."""
+        if not self.is_running():
+            return (
+                "Ollama is not running.\n"
+                "  Start it with:  ollama serve\n"
+                "  Or install:     brew install ollama"
+            )
+        if not self.is_model_available():
+            return (
+                f"Model '{self.model}' is not pulled.\n"
+                f"  Pull it with:   ollama pull {self.model}"
+            )
+        return None
+
+    # -- Chat -----------------------------------------------------------------
+
+    def chat_stream(
+        self,
+        messages: list[dict],
+        temperature: float = LLM_TEMPERATURE,
+        max_tokens: int = LLM_MAX_TOKENS,
+        top_p: float = LLM_TOP_P,
+    ) -> Generator[str, None, None]:
+        """
+        Send a chat completion request and yield tokens as they arrive.
+        `messages` follows the OpenAI-style format:
+            [{"role": "system", "content": "..."}, {"role": "user", "content": "..."}]
+        """
+        payload = {
+            "model": self.model,
+            "messages": messages,
+            "stream": True,
+            "options": {
+                "temperature": temperature,
+                "num_predict": max_tokens,
+                "top_p": top_p,
+            },
+        }
+
+        try:
+            resp = requests.post(
+                f"{self.base_url}/api/chat",
+                json=payload,
+                stream=True,
+                timeout=120,
+            )
+            resp.raise_for_status()
+        except requests.ConnectionError:
+            raise OllamaError(
+                "Cannot reach Ollama. Is it running? (ollama serve)"
+            )
+        except requests.HTTPError as e:
+            raise OllamaError(f"Ollama returned an error: {e}")
+
+        for line in resp.iter_lines(decode_unicode=True):
+            if not line:
+                continue
+            try:
+                chunk = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+
+            token = chunk.get("message", {}).get("content", "")
+            if token:
+                yield token
+
+            if chunk.get("done", False):
+                return
+
+    def chat(self, messages: list[dict], **kwargs) -> str:
+        """Non-streaming convenience wrapper."""
+        return "".join(self.chat_stream(messages, **kwargs))

chatbot/llm_groq.py

@@ -0,0 +1,72 @@
+"""
+Groq API client for LLM chat completions.
+
+Uses the Groq SDK with streaming support. Drop-in replacement for OllamaClient
+when deploying to environments without a local Ollama server.
+"""
+
+from typing import Generator
+
+from groq import Groq
+
+from chatbot.config import (
+    GROQ_API_KEY,
+    GROQ_MODEL,
+    LLM_TEMPERATURE,
+    LLM_MAX_TOKENS,
+    LLM_TOP_P,
+)
+
+
+class GroqError(Exception):
+    pass
+
+
+class GroqClient:
+    def __init__(self, api_key: str = GROQ_API_KEY, model: str = GROQ_MODEL):
+        if not api_key:
+            raise GroqError(
+                "GROQ_API_KEY is not set. Get a free key at https://console.groq.com/keys"
+            )
+        self._client = Groq(api_key=api_key)
+        self.model = model
+
+    def check_ready(self) -> str | None:
+        """Return an error message if not ready, else None."""
+        try:
+            self._client.models.list()
+            return None
+        except Exception as e:
+            return f"Groq API error: {e}"
+
+    def chat_stream(
+        self,
+        messages: list[dict],
+        temperature: float = LLM_TEMPERATURE,
+        max_tokens: int = LLM_MAX_TOKENS,
+        top_p: float = LLM_TOP_P,
+    ) -> Generator[str, None, None]:
+        try:
+            stream = self._client.chat.completions.create(
+                model=self.model,
+                messages=messages,
+                temperature=temperature,
+                max_completion_tokens=max_tokens,
+                top_p=top_p,
+                stream=True,
+            )
+            for chunk in stream:
+                delta = chunk.choices[0].delta
+                if delta and delta.content:
+                    yield delta.content
+        except Exception as e:
+            error_msg = str(e).lower()
+            if "rate_limit" in error_msg or "429" in error_msg:
+                raise GroqError(
+                    "Groq rate limit reached. Please wait a moment and try again."
+                )
+            raise GroqError(f"Groq API error: {e}")
+
+    def chat(self, messages: list[dict], **kwargs) -> str:
+        """Non-streaming convenience wrapper."""
+        return "".join(self.chat_stream(messages, **kwargs))

chatbot/prompts.py

@@ -0,0 +1,89 @@
+"""
+System prompt, context formatting, and chunk deduplication for the RAG chatbot.
+"""
+
+from chatbot.config import MAX_CONTEXT_CHARS, MAX_CHUNK_CHARS
+from chatbot.retriever import ChunkResult
+
+SYSTEM_PROMPT = """\
+You are a UHC medical policy assistant. Users are doctors and hospital staff.
+
+RULES:
+1. Answer ONLY from the policy excerpts below. No outside knowledge.
+2. Be BRIEF: 2-4 bullet points max. One short paragraph for summary if needed.
+3. Cite sources as (policy-name, Section). Example: (bariatric-surgery, Coverage Rationale).
+4. If context lacks the answer, say "I don't have enough policy information to answer this."
+5. If something is "unproven and not medically necessary," say it is NOT covered.
+6. Do NOT repeat or paraphrase the same point multiple times. State each fact once.
+7. Coverage Rationale is the authoritative source for what IS and IS NOT covered. \
+If Coverage Rationale says a treatment is unproven/not medically necessary for a condition, \
+clearly state it is NOT covered — even if Clinical Evidence discusses studies about it."""
+
+
+def deduplicate_chunks(chunks: list[ChunkResult]) -> list[ChunkResult]:
+    """Keep the highest-scoring chunk per (policy_name, section) pair."""
+    seen: dict[tuple[str, str], ChunkResult] = {}
+    for c in chunks:
+        key = (c.policy_name, c.section)
+        if key not in seen or c.score > seen[key].score:
+            seen[key] = c
+    return sorted(seen.values(), key=lambda c: c.score, reverse=True)
+
+
+def _truncate_text(text: str, max_chars: int = MAX_CHUNK_CHARS) -> str:
+    """Truncate chunk text to max_chars, breaking at sentence boundary."""
+    if len(text) <= max_chars:
+        return text
+    cut = text[:max_chars]
+    last_period = cut.rfind(". ")
+    if last_period > max_chars // 2:
+        return cut[:last_period + 1]
+    return cut + "..."
+
+
+def format_context(chunks: list[ChunkResult]) -> str:
+    """
+    Render retrieved chunks into a numbered context block for the LLM.
+    Each chunk is truncated to MAX_CHUNK_CHARS, total capped at MAX_CONTEXT_CHARS.
+    """
+    deduped = deduplicate_chunks(chunks)
+
+    parts = []
+    char_count = 0
+
+    for i, c in enumerate(deduped, 1):
+        header = f"[{i}] {c.policy_name} | {c.section} | {c.plan_type}"
+        body = _truncate_text(c.text)
+        block = f"{header}\n{body}\n"
+
+        if char_count + len(block) > MAX_CONTEXT_CHARS:
+            break
+
+        parts.append(block)
+        char_count += len(block)
+
+    return "\n---\n".join(parts)
+
+
+def build_messages(
+    query: str,
+    context: str,
+    history: list[dict] | None = None,
+) -> list[dict]:
+    """
+    Build the Ollama chat messages list.
+    `history` is a list of prior {"role": ..., "content": ...} entries.
+    """
+    messages = [{"role": "system", "content": SYSTEM_PROMPT}]
+
+    if history:
+        messages.extend(history)
+
+    user_content = (
+        f"CONTEXT:\n{context}\n\n"
+        f"QUESTION: {query}\n\n"
+        f"Answer briefly in 2-4 bullet points with citations."
+    )
+    messages.append({"role": "user", "content": user_content})
+
+    return messages

chatbot/retriever.py

@@ -0,0 +1,169 @@
+"""
+Retriever wrapping the embedding pipeline.
+
+Loads MedEmbed + Qdrant client once and exposes a simple `retrieve(query)` interface.
+Filters out low-value sections (References, Application) that pollute results.
+"""
+
+import sys
+import time
+from dataclasses import dataclass
+
+import torch
+from sentence_transformers import SentenceTransformer
+from qdrant_client import QdrantClient
+from qdrant_client.models import (
+    Filter,
+    FieldCondition,
+    MatchValue,
+    MatchExcept,
+)
+
+from chatbot.config import (
+    EMBEDDING_SCRIPTS_DIR,
+    RETRIEVAL_TOP_K,
+    EXCLUDED_SECTIONS,
+)
+
+sys.path.insert(0, str(EMBEDDING_SCRIPTS_DIR))
+from config import (
+    EMBEDDING_MODEL_NAME,
+    MAX_SEQ_LENGTH,
+    QDRANT_URL,
+    QDRANT_API_KEY,
+    QDRANT_HOST,
+    QDRANT_PORT,
+    QDRANT_COLLECTION,
+)
+
+MAX_RETRIES = 3
+RETRY_BACKOFF = 2
+
+
+@dataclass
+class ChunkResult:
+    text: str
+    policy_name: str
+    section: str
+    plan_type: str
+    score: float
+    page_start: int = 0
+    page_end: int = 0
+
+
+class PolicyRetriever:
+    """Loads models once, reuses across queries."""
+
+    def __init__(self):
+        self._model = None
+        self._device = None
+        self._client = None
+
+    def _ensure_model(self):
+        if self._model is not None:
+            return
+        self._device = (
+            "cuda" if torch.cuda.is_available()
+            else "mps" if torch.backends.mps.is_available()
+            else "cpu"
+        )
+        self._model = SentenceTransformer(EMBEDDING_MODEL_NAME, trust_remote_code=False)
+        self._model.max_seq_length = MAX_SEQ_LENGTH
+
+    def _ensure_client(self):
+        if self._client is not None:
+            return
+        if QDRANT_URL:
+            self._client = QdrantClient(
+                url=QDRANT_URL, api_key=QDRANT_API_KEY,
+                timeout=30, prefer_grpc=False,
+            )
+        else:
+            self._client = QdrantClient(host=QDRANT_HOST, port=QDRANT_PORT, timeout=30)
+
+    def init(self, status_callback=None):
+        """Eagerly load model + client. Optional callback for progress messages."""
+        cb = status_callback or (lambda msg: None)
+
+        cb("Loading MedEmbed model...")
+        self._ensure_model()
+        cb(f"  Model loaded on {self._device}")
+
+        cb("Connecting to Qdrant...")
+        self._ensure_client()
+        cb("  Connected")
+        return self
+
+    def retrieve(
+        self,
+        query: str,
+        top_k: int = RETRIEVAL_TOP_K,
+        section_filter: str | None = None,
+        policy_filter: str | None = None,
+        exclude_sections: bool = True,
+    ) -> list[ChunkResult]:
+        self._ensure_model()
+        self._ensure_client()
+
+        vec = self._model.encode(
+            query,
+            convert_to_numpy=True,
+            normalize_embeddings=True,
+            device=self._device,
+        ).tolist()
+
+        conditions = []
+        if exclude_sections and EXCLUDED_SECTIONS:
+            conditions.append(
+                FieldCondition(
+                    key="section",
+                    match=MatchExcept(**{"except": list(EXCLUDED_SECTIONS)}),
+                )
+            )
+        if section_filter:
+            conditions.append(FieldCondition(key="section", match=MatchValue(value=section_filter)))
+        if policy_filter:
+            conditions.append(FieldCondition(key="policy_name", match=MatchValue(value=policy_filter)))
+
+        qf = Filter(must=conditions) if conditions else None
+
+        for attempt in range(1, MAX_RETRIES + 1):
+            try:
+                hits = self._client.query_points(
+                    collection_name=QDRANT_COLLECTION,
+                    query=vec,
+                    query_filter=qf,
+                    limit=top_k,
+                    with_payload=True,
+                ).points
+                break
+            except Exception as e:
+                if attempt < MAX_RETRIES:
+                    time.sleep(RETRY_BACKOFF ** attempt)
+                else:
+                    raise RuntimeError(
+                        f"Qdrant query failed after {MAX_RETRIES} retries: {e}"
+                    ) from e
+
+        results = []
+        for hit in hits:
+            p = hit.payload
+            results.append(ChunkResult(
+                text=p.get("text", ""),
+                policy_name=p.get("policy_name", ""),
+                section=p.get("section", ""),
+                plan_type=p.get("plan_type", ""),
+                score=hit.score,
+                page_start=p.get("page_start", 0),
+                page_end=p.get("page_end", 0),
+            ))
+
+        # Boost Coverage Rationale so it always survives deduplication.
+        # This ensures the authoritative coverage/non-coverage statement
+        # is present even when Clinical Evidence chunks score higher.
+        COVERAGE_BOOST = 0.02
+        for r in results:
+            if r.section == "Coverage Rationale":
+                r.score += COVERAGE_BOOST
+
+        return results

embedding/scripts/config.py

@@ -0,0 +1,33 @@
+import os
+from pathlib import Path
+from dotenv import load_dotenv
+
+load_dotenv(Path(__file__).resolve().parent.parent.parent / ".env")
+
+BASE_DIR = Path(__file__).resolve().parent.parent
+PROJECT_ROOT = BASE_DIR.parent
+
+# --- Paths ---
+RAG_CHUNKS_PATH = PROJECT_ROOT / "scraper" / "data" / "processed" / "rag_chunks.json"
+EMBEDDINGS_DIR = BASE_DIR / "data" / "embeddings"
+EMBEDDINGS_FILE = EMBEDDINGS_DIR / "chunk_embeddings.npz"
+
+# --- Embedding Model ---
+EMBEDDING_MODEL_NAME = "abhinand/MedEmbed-large-v0.1"
+EMBEDDING_DIM = 1024
+BATCH_SIZE = 32
+MAX_SEQ_LENGTH = 512
+
+# --- Qdrant ---
+QDRANT_URL = os.getenv("QDRANT_URL", None)
+QDRANT_API_KEY = os.getenv("QDRANT_API_KEY", None)
+QDRANT_HOST = os.getenv("QDRANT_HOST", "localhost")
+QDRANT_PORT = int(os.getenv("QDRANT_PORT", "6333"))
+QDRANT_COLLECTION = os.getenv("QDRANT_COLLECTION", "uhc_policies")
+
+# --- Search ---
+TOP_K = 10
+
+# --- Provider (for multi-provider extensibility) ---
+PROVIDER_NAME = "UnitedHealthcare"
+PROVIDER_SLUG = "uhc"

embedding/scripts/embed_chunks.py

@@ -0,0 +1,128 @@
+"""
+Generate embeddings for RAG chunks using MedEmbed-large-v0.1.
+
+Reads rag_chunks.json, encodes every chunk's text with the MedEmbed model,
+and saves the resulting vectors alongside their chunk IDs to a compressed
+numpy archive (.npz) for downstream loading into Qdrant.
+
+Usage:
+    python embed_chunks.py                      # full run
+    python embed_chunks.py --limit 100          # embed only first 100 chunks (for testing)
+    python embed_chunks.py --batch-size 64      # override batch size
+"""
+
+import argparse
+import json
+import time
+import numpy as np
+import torch
+from sentence_transformers import SentenceTransformer
+from tqdm import tqdm
+
+from config import (
+    RAG_CHUNKS_PATH,
+    EMBEDDINGS_DIR,
+    EMBEDDINGS_FILE,
+    EMBEDDING_MODEL_NAME,
+    BATCH_SIZE,
+    MAX_SEQ_LENGTH,
+)
+
+
+def select_device() -> str:
+    if torch.cuda.is_available():
+        return "cuda"
+    if torch.backends.mps.is_available():
+        return "mps"
+    return "cpu"
+
+
+def load_chunks(path, limit=None):
+    with open(path, "r", encoding="utf-8") as f:
+        chunks = json.load(f)
+    if limit:
+        chunks = chunks[:limit]
+    return chunks
+
+
+def build_embedding_text(chunk: dict) -> str:
+    """
+    Construct the text that gets embedded. Prepend key metadata so the
+    embedding captures policy context, not just the raw paragraph.
+    """
+    parts = []
+
+    policy = chunk.get("policy_name", "").replace("-", " ").title()
+    if policy:
+        parts.append(f"Policy: {policy}")
+
+    section = chunk.get("section", "")
+    if section:
+        parts.append(f"Section: {section}")
+
+    parts.append(chunk["text"])
+
+    return " | ".join(parts)
+
+
+def embed_in_batches(model, texts, batch_size, device):
+    all_embeddings = []
+
+    for i in tqdm(range(0, len(texts), batch_size), desc="Embedding batches"):
+        batch = texts[i : i + batch_size]
+        embeddings = model.encode(
+            batch,
+            batch_size=batch_size,
+            show_progress_bar=False,
+            convert_to_numpy=True,
+            normalize_embeddings=True,
+            device=device,
+        )
+        all_embeddings.append(embeddings)
+
+    return np.vstack(all_embeddings)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Embed RAG chunks with MedEmbed")
+    parser.add_argument("--limit", type=int, default=None, help="Limit chunks to embed (for testing)")
+    parser.add_argument("--batch-size", type=int, default=BATCH_SIZE, help="Batch size for encoding")
+    args = parser.parse_args()
+
+    device = select_device()
+    print(f"Device: {device}")
+    print(f"Model:  {EMBEDDING_MODEL_NAME}")
+    print(f"Batch:  {args.batch_size}")
+
+    print("\nLoading chunks...")
+    chunks = load_chunks(RAG_CHUNKS_PATH, limit=args.limit)
+    print(f"Loaded {len(chunks)} chunks")
+
+    chunk_ids = [c["id"] for c in chunks]
+    texts = [build_embedding_text(c) for c in chunks]
+
+    print(f"\nLoading model {EMBEDDING_MODEL_NAME}...")
+    model = SentenceTransformer(EMBEDDING_MODEL_NAME, trust_remote_code=True)
+    model.max_seq_length = MAX_SEQ_LENGTH
+    print(f"Model loaded — embedding dim: {model.get_sentence_embedding_dimension()}")
+
+    print("\nGenerating embeddings...")
+    start = time.time()
+    embeddings = embed_in_batches(model, texts, args.batch_size, device)
+    elapsed = time.time() - start
+
+    print(f"\nEmbeddings shape: {embeddings.shape}")
+    print(f"Time: {elapsed:.1f}s ({len(texts) / elapsed:.1f} chunks/sec)")
+
+    EMBEDDINGS_DIR.mkdir(parents=True, exist_ok=True)
+    np.savez_compressed(
+        EMBEDDINGS_FILE,
+        ids=np.array(chunk_ids, dtype=object),
+        embeddings=embeddings,
+    )
+    size_mb = EMBEDDINGS_FILE.stat().st_size / (1024 * 1024)
+    print(f"\nSaved to {EMBEDDINGS_FILE} ({size_mb:.1f} MB)")
+
+
+if __name__ == "__main__":
+    main()

embedding/scripts/search.py

@@ -0,0 +1,155 @@
+"""
+Test search interface against the Qdrant vector store.
+
+Encodes a query with the same MedEmbed model and retrieves the top-K
+most relevant policy chunks, demonstrating the full retrieval pipeline.
+
+Usage:
+    python search.py "Is bariatric surgery covered for BMI over 40?"
+    python search.py "What CPT codes are used for cochlear implants?"
+    python search.py "criteria for sleep apnea treatment" --top-k 5
+    python search.py "coverage for gene therapy hemophilia" --section "Coverage Rationale"
+"""
+
+import argparse
+import sys
+import time
+
+import torch
+from sentence_transformers import SentenceTransformer
+from qdrant_client import QdrantClient
+from qdrant_client.models import Filter, FieldCondition, MatchValue
+
+from config import (
+    EMBEDDING_MODEL_NAME,
+    MAX_SEQ_LENGTH,
+    QDRANT_HOST,
+    QDRANT_PORT,
+    QDRANT_COLLECTION,
+    QDRANT_URL,
+    QDRANT_API_KEY,
+    TOP_K,
+)
+
+MAX_RETRIES = 3
+RETRY_BACKOFF = 2
+
+
+def get_client() -> QdrantClient:
+    if QDRANT_URL:
+        if not QDRANT_API_KEY or QDRANT_API_KEY == "YOUR_API_KEY_HERE":
+            print(
+                "WARNING: QDRANT_API_KEY is not set or still a placeholder.\n"
+                "  Filtered queries WILL fail. Set a real key in .env",
+                file=sys.stderr,
+            )
+        return QdrantClient(
+            url=QDRANT_URL,
+            api_key=QDRANT_API_KEY,
+            timeout=30,
+            prefer_grpc=False,
+        )
+    return QdrantClient(host=QDRANT_HOST, port=QDRANT_PORT, timeout=30)
+
+
+def load_model():
+    device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
+    model = SentenceTransformer(EMBEDDING_MODEL_NAME, trust_remote_code=False)
+    model.max_seq_length = MAX_SEQ_LENGTH
+    return model, device
+
+
+def search(client, model, device, query, top_k=TOP_K, section_filter=None, policy_filter=None):
+    query_vector = model.encode(
+        query,
+        convert_to_numpy=True,
+        normalize_embeddings=True,
+        device=device,
+    ).tolist()
+
+    conditions = []
+    if section_filter:
+        conditions.append(FieldCondition(key="section", match=MatchValue(value=section_filter)))
+    if policy_filter:
+        conditions.append(FieldCondition(key="policy_name", match=MatchValue(value=policy_filter)))
+
+    search_filter = Filter(must=conditions) if conditions else None
+
+    for attempt in range(1, MAX_RETRIES + 1):
+        try:
+            results = client.query_points(
+                collection_name=QDRANT_COLLECTION,
+                query=query_vector,
+                query_filter=search_filter,
+                limit=top_k,
+                with_payload=True,
+            )
+            return results.points
+        except Exception as e:
+            if attempt < MAX_RETRIES:
+                wait = RETRY_BACKOFF ** attempt
+                print(f"  Connection error (attempt {attempt}/{MAX_RETRIES}), retrying in {wait}s...")
+                time.sleep(wait)
+            else:
+                raise RuntimeError(
+                    f"Failed after {MAX_RETRIES} attempts. Last error: {e}\n"
+                    "Check that QDRANT_URL and QDRANT_API_KEY in .env are correct."
+                ) from e
+
+
+def format_result(hit, rank):
+    p = hit.payload
+    lines = [
+        f"\n{'='*80}",
+        f"  Rank #{rank}  |  Score: {hit.score:.4f}",
+        f"  Policy:    {p.get('policy_name', 'N/A')}",
+        f"  Section:   {p.get('section', 'N/A')}",
+        f"  Effective: {p.get('effective_date', 'N/A')}",
+        f"  Plan:      {p.get('plan_type', 'N/A')}",
+        f"  Pages:     {p.get('page_start', '?')}-{p.get('page_end', '?')}",
+        f"{'─'*80}",
+    ]
+    text = p.get("text", "")
+    preview = text[:500] + ("..." if len(text) > 500 else "")
+    lines.append(f"  {preview}")
+    lines.append(f"{'='*80}")
+    return "\n".join(lines)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Search UHC policy chunks")
+    parser.add_argument("query", type=str, help="Search query")
+    parser.add_argument("--top-k", type=int, default=TOP_K, help="Number of results")
+    parser.add_argument("--section", type=str, default=None, help="Filter by section name")
+    parser.add_argument("--policy", type=str, default=None, help="Filter by policy slug")
+    args = parser.parse_args()
+
+    print(f"Query: \"{args.query}\"")
+    if args.section:
+        print(f"Section filter: {args.section}")
+    if args.policy:
+        print(f"Policy filter: {args.policy}")
+
+    print("\nLoading model...")
+    model, device = load_model()
+
+    print("Connecting to Qdrant...")
+    client = get_client()
+
+    print(f"Searching (top-{args.top_k})...\n")
+    try:
+        results = search(client, model, device, args.query, args.top_k, args.section, args.policy)
+    except RuntimeError as e:
+        print(f"\nERROR: {e}", file=sys.stderr)
+        sys.exit(1)
+
+    if not results:
+        print("No results found.")
+        return
+
+    for i, hit in enumerate(results, 1):
+        print(format_result(hit, i))
+
+
+if __name__ == "__main__":
+    main()

embedding/scripts/store_qdrant.py

@@ -0,0 +1,170 @@
+"""
+Load precomputed embeddings + chunk metadata and upsert into Qdrant.
+
+Supports both local Qdrant (docker) and Qdrant Cloud via env vars.
+Creates the collection with proper HNSW config if it doesn't exist.
+
+Usage:
+    python store_qdrant.py                  # full upsert
+    python store_qdrant.py --recreate       # drop + recreate collection first
+"""
+
+import argparse
+import json
+import numpy as np
+from tqdm import tqdm
+from qdrant_client import QdrantClient
+from qdrant_client.models import (
+    Distance,
+    VectorParams,
+    PointStruct,
+    HnswConfigDiff,
+    OptimizersConfigDiff,
+    PayloadSchemaType,
+)
+
+from config import (
+    RAG_CHUNKS_PATH,
+    EMBEDDINGS_FILE,
+    EMBEDDING_DIM,
+    QDRANT_HOST,
+    QDRANT_PORT,
+    QDRANT_COLLECTION,
+    QDRANT_URL,
+    QDRANT_API_KEY,
+    PROVIDER_NAME,
+    PROVIDER_SLUG,
+)
+
+UPSERT_BATCH_SIZE = 100
+
+
+def get_client() -> QdrantClient:
+    if QDRANT_URL:
+        return QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY, timeout=60)
+    return QdrantClient(host=QDRANT_HOST, port=QDRANT_PORT, timeout=60)
+
+
+def ensure_collection(client: QdrantClient, recreate: bool = False):
+    exists = client.collection_exists(QDRANT_COLLECTION)
+
+    if exists and recreate:
+        print(f"Dropping existing collection '{QDRANT_COLLECTION}'...")
+        client.delete_collection(QDRANT_COLLECTION)
+        exists = False
+
+    if not exists:
+        print(f"Creating collection '{QDRANT_COLLECTION}' (dim={EMBEDDING_DIM})...")
+        client.create_collection(
+            collection_name=QDRANT_COLLECTION,
+            vectors_config=VectorParams(
+                size=EMBEDDING_DIM,
+                distance=Distance.COSINE,
+                on_disk=False,
+            ),
+            hnsw_config=HnswConfigDiff(
+                m=16,
+                ef_construct=100,
+            ),
+            optimizers_config=OptimizersConfigDiff(
+                indexing_threshold=20000,
+            ),
+        )
+        print("Collection created.")
+
+    print("Ensuring payload indexes for filtered search...")
+    for field in ("section", "policy_name", "plan_type", "doc_type", "provider"):
+        client.create_payload_index(
+            collection_name=QDRANT_COLLECTION,
+            field_name=field,
+            field_schema=PayloadSchemaType.KEYWORD,
+        )
+    print("  Indexes created: section, policy_name, plan_type, doc_type, provider")
+
+
+def load_data():
+    print("Loading embeddings...")
+    data = np.load(EMBEDDINGS_FILE, allow_pickle=True)
+    ids = data["ids"]
+    embeddings = data["embeddings"]
+    print(f"  Loaded {len(ids)} embeddings of dim {embeddings.shape[1]}")
+
+    print("Loading chunk metadata...")
+    with open(RAG_CHUNKS_PATH, "r", encoding="utf-8") as f:
+        chunks = json.load(f)
+
+    chunk_map = {c["id"]: c for c in chunks}
+    print(f"  Loaded {len(chunks)} chunks")
+
+    return ids, embeddings, chunk_map
+
+
+def build_payload(chunk: dict) -> dict:
+    return {
+        "policy_name": chunk.get("policy_name", ""),
+        "policy_number": chunk.get("policy_number", ""),
+        "effective_date": chunk.get("effective_date", ""),
+        "plan_type": chunk.get("plan_type", ""),
+        "doc_type": chunk.get("doc_type", ""),
+        "section": chunk.get("section", ""),
+        "page_start": chunk.get("page_start", 0),
+        "page_end": chunk.get("page_end", 0),
+        "chunk_index": chunk.get("chunk_index", 0),
+        "total_chunks_in_section": chunk.get("total_chunks_in_section", 0),
+        "text": chunk.get("text", ""),
+        "provider": PROVIDER_SLUG,
+    }
+
+
+def upsert_points(client, ids, embeddings, chunk_map):
+    points = []
+    skipped = 0
+
+    for i, (chunk_id, vector) in enumerate(zip(ids, embeddings)):
+        chunk_id_str = str(chunk_id)
+        if chunk_id_str not in chunk_map:
+            skipped += 1
+            continue
+
+        payload = build_payload(chunk_map[chunk_id_str])
+
+        points.append(
+            PointStruct(
+                id=i,
+                vector=vector.tolist(),
+                payload=payload,
+            )
+        )
+
+    if skipped:
+        print(f"  Skipped {skipped} embeddings (no matching chunk metadata)")
+
+    print(f"  Upserting {len(points)} points in batches of {UPSERT_BATCH_SIZE}...")
+
+    for batch_start in tqdm(range(0, len(points), UPSERT_BATCH_SIZE), desc="Upserting"):
+        batch = points[batch_start : batch_start + UPSERT_BATCH_SIZE]
+        client.upsert(collection_name=QDRANT_COLLECTION, points=batch, wait=True)
+
+    return len(points)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Store embeddings in Qdrant")
+    parser.add_argument("--recreate", action="store_true", help="Drop and recreate collection")
+    args = parser.parse_args()
+
+    client = get_client()
+    ensure_collection(client, recreate=args.recreate)
+
+    ids, embeddings, chunk_map = load_data()
+    total = upsert_points(client, ids, embeddings, chunk_map)
+
+    info = client.get_collection(QDRANT_COLLECTION)
+    print(f"\nDone. Collection '{QDRANT_COLLECTION}' now has {info.points_count} points.")
+    print(f"  Vectors dim: {EMBEDDING_DIM}")
+    print(f"  Distance:    COSINE")
+    print(f"  Provider:    {PROVIDER_NAME}")
+
+
+if __name__ == "__main__":
+    main()

embedding/scripts/test_retrieval.py

@@ -0,0 +1,386 @@
+"""
+Batch retrieval test suite — loads the model once and runs 10 test queries
+against Qdrant, graded by difficulty (4 easy, 3 medium, 3 hard).
+
+Each test specifies:
+  - query: natural-language question a doctor/staff might ask
+  - expected_policy: slug that MUST appear in the top-K results
+  - expected_section: section that SHOULD appear for the best hit
+  - filters: optional section/policy filter to exercise filtered search
+  - difficulty: easy | medium | hard
+
+Scoring:
+  - policy_hit@K  : expected policy appears anywhere in top-K
+  - policy_hit@1  : expected policy is the rank-1 result
+  - section_match : rank-1 result matches expected section
+  - mrr           : 1/rank of first correct-policy hit (mean reciprocal rank)
+
+Usage:
+    python test_retrieval.py
+    python test_retrieval.py --top-k 5
+    python test_retrieval.py --verbose
+"""
+
+import argparse
+import json
+import sys
+import time
+from dataclasses import dataclass, field
+
+import torch
+from sentence_transformers import SentenceTransformer
+from qdrant_client import QdrantClient
+from qdrant_client.models import Filter, FieldCondition, MatchValue
+
+from config import (
+    EMBEDDING_MODEL_NAME,
+    MAX_SEQ_LENGTH,
+    QDRANT_HOST,
+    QDRANT_PORT,
+    QDRANT_COLLECTION,
+    QDRANT_URL,
+    QDRANT_API_KEY,
+    TOP_K,
+)
+
+# ── Test cases ────────────────────────────────────────────────────────────────
+
+TEST_CASES = [
+    # ── EASY (4): direct keyword overlap, single policy, obvious answer ──────
+    {
+        "id": "E1",
+        "difficulty": "easy",
+        "query": "Is bariatric surgery covered for patients with BMI over 40?",
+        "expected_policy": "bariatric-surgery",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Direct policy name in query; BMI 40 threshold explicitly stated in Coverage Rationale.",
+    },
+    {
+        "id": "E2",
+        "difficulty": "easy",
+        "query": "What conditions are treated with hyperbaric oxygen therapy?",
+        "expected_policy": "hyperbaric-topical-oxygen-therapy",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Policy lists conditions (crush injury, osteomyelitis, etc.) directly in Coverage Rationale.",
+    },
+    {
+        "id": "E3",
+        "difficulty": "easy",
+        "query": "What is the coverage policy for cochlear implants in adults?",
+        "expected_policy": "cochlear-implants",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Exact policy name; Coverage Rationale states criteria for adults 18+.",
+    },
+    {
+        "id": "E4",
+        "difficulty": "easy",
+        "query": "Is TENS covered for pain management?",
+        "expected_policy": "electrical-stimulation-treatment-pain-muscle-rehabilitation",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "TENS is the primary device discussed in this policy's Coverage Rationale.",
+    },
+
+    # ── MEDIUM (3): requires semantic understanding, cross-section, or filter ─
+    {
+        "id": "M1",
+        "difficulty": "medium",
+        "query": "What are the eligibility criteria for gene therapy in hemophilia B patients?",
+        "expected_policy": "gene-therapies-hemophilia",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Must match 'hemophilia B' to Beqvez criteria; query uses 'eligibility' not 'coverage'.",
+    },
+    {
+        "id": "M2",
+        "difficulty": "medium",
+        "query": "When is proton beam radiation approved instead of standard radiation for cancer?",
+        "expected_policy": "proton-beam-radiation-therapy",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Requires understanding that PBRT is an alternative; policy specifies indications by age and tumor type.",
+    },
+    {
+        "id": "M3",
+        "difficulty": "medium",
+        "query": "Does UHC cover continuous glucose monitors for diabetic patients on insulin pumps?",
+        "expected_policy": "continuous-glucose-monitoring-insulin-delivery-managing-diabetes",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Long policy slug; query combines two sub-topics (CGM + insulin delivery) from the same policy.",
+    },
+
+    # ── HARD (3): paraphrased, multi-hop, or requires domain reasoning ────────
+    {
+        "id": "H1",
+        "difficulty": "hard",
+        "query": "A 16-year-old patient needs genetic testing for an undiagnosed developmental disorder — is whole genome sequencing covered?",
+        "expected_policy": "whole-exome-and-whole-genome-sequencing",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Heavily paraphrased; must link 'undiagnosed developmental disorder' + 'genetic testing' to WES/WGS policy criteria about suspected genetic cause.",
+    },
+    {
+        "id": "H2",
+        "difficulty": "hard",
+        "query": "What documentation is needed before a patient can get gender-affirming mastectomy?",
+        "expected_policy": "gender-dysphoria-treatment",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Uses 'gender-affirming mastectomy' instead of 'Gender Dysphoria'; must connect to breast surgery documentation requirements in Coverage Rationale.",
+    },
+    {
+        "id": "H3",
+        "difficulty": "hard",
+        "query": "Patient has failed oral appliance therapy for sleep apnea — what surgical options does UHC cover?",
+        "expected_policy": "obstructive-sleep-apnea-treatment",
+        "expected_section": "Coverage Rationale",
+        "filters": {},
+        "rationale": "Multi-hop reasoning: failed OAT → surgical alternatives; query never uses 'obstructive' or policy name. Must infer from clinical scenario.",
+    },
+]
+
+
+# ── Helpers ───────────────────────────────────────────────────────────────────
+
+MAX_RETRIES = 3
+RETRY_BACKOFF = 2
+
+PASS = "\033[92m✓ PASS\033[0m"
+FAIL = "\033[91m✗ FAIL\033[0m"
+WARN = "\033[93m~ PARTIAL\033[0m"
+
+
+def get_client() -> QdrantClient:
+    if QDRANT_URL:
+        return QdrantClient(url=QDRANT_URL, api_key=QDRANT_API_KEY, timeout=30, prefer_grpc=False)
+    return QdrantClient(host=QDRANT_HOST, port=QDRANT_PORT, timeout=30)
+
+
+def run_query(client, model, device, query, top_k, section_filter=None, policy_filter=None):
+    vec = model.encode(query, convert_to_numpy=True, normalize_embeddings=True, device=device).tolist()
+
+    conditions = []
+    if section_filter:
+        conditions.append(FieldCondition(key="section", match=MatchValue(value=section_filter)))
+    if policy_filter:
+        conditions.append(FieldCondition(key="policy_name", match=MatchValue(value=policy_filter)))
+
+    qf = Filter(must=conditions) if conditions else None
+
+    for attempt in range(1, MAX_RETRIES + 1):
+        try:
+            return client.query_points(
+                collection_name=QDRANT_COLLECTION,
+                query=vec,
+                query_filter=qf,
+                limit=top_k,
+                with_payload=True,
+            ).points
+        except Exception as e:
+            if attempt < MAX_RETRIES:
+                time.sleep(RETRY_BACKOFF ** attempt)
+            else:
+                raise RuntimeError(f"Qdrant query failed after {MAX_RETRIES} retries: {e}") from e
+
+
+@dataclass
+class TestResult:
+    test_id: str
+    difficulty: str
+    query: str
+    expected_policy: str
+    expected_section: str
+    policy_hit_at_k: bool = False
+    policy_hit_at_1: bool = False
+    section_match: bool = False
+    first_hit_rank: int = 0
+    top1_policy: str = ""
+    top1_section: str = ""
+    top1_score: float = 0.0
+    latency_ms: float = 0.0
+    error: str = ""
+
+
+def evaluate(tc: dict, client, model, device, top_k: int) -> TestResult:
+    res = TestResult(
+        test_id=tc["id"],
+        difficulty=tc["difficulty"],
+        query=tc["query"],
+        expected_policy=tc["expected_policy"],
+        expected_section=tc["expected_section"],
+    )
+
+    t0 = time.perf_counter()
+    try:
+        hits = run_query(
+            client, model, device,
+            tc["query"], top_k,
+            tc["filters"].get("section"),
+            tc["filters"].get("policy"),
+        )
+    except RuntimeError as e:
+        res.error = str(e)
+        res.latency_ms = (time.perf_counter() - t0) * 1000
+        return res
+
+    res.latency_ms = (time.perf_counter() - t0) * 1000
+
+    if not hits:
+        return res
+
+    res.top1_policy = hits[0].payload.get("policy_name", "")
+    res.top1_section = hits[0].payload.get("section", "")
+    res.top1_score = hits[0].score
+
+    res.policy_hit_at_1 = res.top1_policy == tc["expected_policy"]
+    res.section_match = res.top1_section == tc["expected_section"]
+
+    for rank, hit in enumerate(hits, 1):
+        if hit.payload.get("policy_name") == tc["expected_policy"]:
+            res.policy_hit_at_k = True
+            res.first_hit_rank = rank
+            break
+
+    return res
+
+
+# ── Main ──────────────────────────────────────────────────────────────────────
+
+def main():
+    parser = argparse.ArgumentParser(description="Batch retrieval test suite")
+    parser.add_argument("--top-k", type=int, default=TOP_K)
+    parser.add_argument("--verbose", "-v", action="store_true", help="Print top-3 results per test")
+    args = parser.parse_args()
+
+    print("=" * 80)
+    print("  UHC Policy RAG — Retrieval Test Suite")
+    print(f"  Model: {EMBEDDING_MODEL_NAME}  |  Top-K: {args.top_k}")
+    print("=" * 80)
+
+    print("\nLoading model (one-time)...")
+    device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
+    model = SentenceTransformer(EMBEDDING_MODEL_NAME, trust_remote_code=False)
+    model.max_seq_length = MAX_SEQ_LENGTH
+    print(f"  Model loaded on {device}")
+
+    print("Connecting to Qdrant...\n")
+    client = get_client()
+
+    results: list[TestResult] = []
+
+    for tc in TEST_CASES:
+        r = evaluate(tc, client, model, device, args.top_k)
+        results.append(r)
+
+        if r.error:
+            status = f"\033[91mERROR\033[0m"
+        elif r.policy_hit_at_1 and r.section_match:
+            status = PASS
+        elif r.policy_hit_at_k:
+            status = WARN
+        else:
+            status = FAIL
+
+        print(f"  [{r.test_id}] {status}  ({r.difficulty.upper():6s})  {r.latency_ms:6.0f}ms  "
+              f"score={r.top1_score:.4f}  {r.query[:60]}...")
+
+        if r.error:
+            print(f"        ERROR: {r.error[:120]}")
+        elif not r.policy_hit_at_1:
+            print(f"        Expected: {r.expected_policy} / {r.expected_section}")
+            print(f"        Got top1: {r.top1_policy} / {r.top1_section}")
+            if r.policy_hit_at_k:
+                print(f"        Correct policy first found at rank #{r.first_hit_rank}")
+
+        if args.verbose and not r.error:
+            hits = run_query(
+                client, model, device,
+                tc["query"], min(3, args.top_k),
+                tc["filters"].get("section"),
+                tc["filters"].get("policy"),
+            )
+            for rank, hit in enumerate(hits, 1):
+                p = hit.payload
+                print(f"          #{rank} [{hit.score:.4f}] {p.get('policy_name')} / {p.get('section')} | {p.get('text')}")
+
+    # ── Summary ───────────────────────────────────────────────────────────
+    print("\n" + "=" * 80)
+    print("  SUMMARY")
+    print("=" * 80)
+
+    valid = [r for r in results if not r.error]
+    errored = [r for r in results if r.error]
+
+    if not valid:
+        print("  All tests errored. Check Qdrant connection / API key.")
+        sys.exit(1)
+
+    hit_at_1 = sum(1 for r in valid if r.policy_hit_at_1)
+    hit_at_k = sum(1 for r in valid if r.policy_hit_at_k)
+    section_ok = sum(1 for r in valid if r.policy_hit_at_1 and r.section_match)
+    mrr = sum((1.0 / r.first_hit_rank) for r in valid if r.first_hit_rank > 0) / len(valid)
+    avg_latency = sum(r.latency_ms for r in valid) / len(valid)
+    avg_score = sum(r.top1_score for r in valid) / len(valid)
+
+    print(f"\n  Total tests:        {len(results)}")
+    print(f"  Successful:         {len(valid)}")
+    if errored:
+        print(f"  Errors:             {len(errored)}  ({', '.join(r.test_id for r in errored)})")
+    print(f"\n  Policy Hit@1:       {hit_at_1}/{len(valid)}  ({100*hit_at_1/len(valid):.0f}%)")
+    print(f"  Policy Hit@K:       {hit_at_k}/{len(valid)}  ({100*hit_at_k/len(valid):.0f}%)")
+    print(f"  Section Match@1:    {section_ok}/{len(valid)}  ({100*section_ok/len(valid):.0f}%)")
+    print(f"  MRR:                {mrr:.4f}")
+    print(f"  Avg Cosine Score:   {avg_score:.4f}")
+    print(f"  Avg Latency:        {avg_latency:.0f}ms")
+
+    for difficulty in ("easy", "medium", "hard"):
+        subset = [r for r in valid if r.difficulty == difficulty]
+        if not subset:
+            continue
+        h1 = sum(1 for r in subset if r.policy_hit_at_1)
+        hk = sum(1 for r in subset if r.policy_hit_at_k)
+        sub_mrr = sum((1.0 / r.first_hit_rank) for r in subset if r.first_hit_rank > 0) / len(subset)
+        print(f"\n  {difficulty.upper():6s}  Hit@1: {h1}/{len(subset)}  Hit@K: {hk}/{len(subset)}  MRR: {sub_mrr:.4f}")
+
+    print("\n" + "=" * 80)
+
+    # ── JSON dump for programmatic use ────────────────────────────────────
+    report = {
+        "model": EMBEDDING_MODEL_NAME,
+        "top_k": args.top_k,
+        "total": len(results),
+        "policy_hit_at_1": hit_at_1,
+        "policy_hit_at_k": hit_at_k,
+        "section_match_at_1": section_ok,
+        "mrr": round(mrr, 4),
+        "avg_cosine_score": round(avg_score, 4),
+        "avg_latency_ms": round(avg_latency, 1),
+        "tests": [
+            {
+                "id": r.test_id,
+                "difficulty": r.difficulty,
+                "query": r.query,
+                "policy_hit_at_1": r.policy_hit_at_1,
+                "policy_hit_at_k": r.policy_hit_at_k,
+                "section_match": r.section_match,
+                "first_hit_rank": r.first_hit_rank,
+                "top1_score": round(r.top1_score, 4),
+                "latency_ms": round(r.latency_ms, 1),
+                "error": r.error or None,
+            }
+            for r in results
+        ],
+    }
+
+    out_path = "test_results.json"
+    with open(out_path, "w") as f:
+        json.dump(report, f, indent=2)
+    print(f"  Results saved to {out_path}\n")
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,24 @@
+# PDF scraping & extraction
+pdfplumber>=0.10.0
+beautifulsoup4>=4.12.0
+requests>=2.31.0
+tqdm>=4.66.0
+pandas>=2.0.0
+
+# Embedding model
+sentence-transformers>=3.0.0
+torch>=2.0.0
+transformers>=4.40.0
+numpy>=1.24.0
+
+# Vector database
+qdrant-client>=1.9.0
+
+# LLM providers
+groq>=0.9.0
+
+# Web UI
+streamlit>=1.35.0
+
+# Utilities
+python-dotenv>=1.0.0

scraper/create_rag_chunks.py

@@ -0,0 +1,278 @@
+import json
+import re
+import hashlib
+from tqdm import tqdm
+
+INPUT_FILE = "data/processed/extracted_sections.json"
+OUTPUT_FILE = "data/processed/rag_chunks.json"
+
+TARGET_CHUNK_TOKENS = 400
+MAX_CHUNK_TOKENS = 600
+OVERLAP_SENTENCES = 2
+
+LOW_VALUE_SECTIONS = {"References", "U.S. Food and Drug Administration"}
+BOILERPLATE_SECTIONS = {"Instructions for Use", "Policy History/Revision Information"}
+
+
+def estimate_tokens(text):
+    return int(len(text.split()) * 1.3)
+
+
+def split_sentences(text):
+    parts = re.split(r'(?<=[.;])\s+(?=[A-Z])', text)
+    sentences = []
+    for part in parts:
+        if estimate_tokens(part) > MAX_CHUNK_TOKENS:
+            sub_parts = re.split(r'(?<=[:;])\s+', part)
+            sentences.extend(sub_parts)
+        else:
+            sentences.append(part)
+    return [s.strip() for s in sentences if s.strip()]
+
+
+def chunk_id(policy, section, idx):
+    raw = f"{policy}__{section}__{idx}"
+    return hashlib.md5(raw.encode()).hexdigest()[:12]
+
+
+def chunk_by_criteria(text, policy_name):
+    blocks = re.split(
+        r'\n\n(?=(?:The following|For (?:initial|continuation|subsequent|revision|replacement)|'
+        r'(?:An?|The)\s+\w.*?is (?:proven|unproven|medically necessary|not medically)|'
+        r'(?:Multiplex|Implantable|Removable|Emergency|Non-Surgical|Surgical)))',
+        text
+    )
+
+    if len(blocks) <= 1:
+        blocks = re.split(r'\n\n', text)
+
+    result = []
+    current = []
+    current_tokens = 0
+
+    for block in blocks:
+        block = block.strip()
+        if not block:
+            continue
+
+        block_tokens = estimate_tokens(block)
+
+        if block_tokens > MAX_CHUNK_TOKENS:
+            if current:
+                result.append("\n\n".join(current))
+                current = []
+                current_tokens = 0
+
+            sents = split_sentences(block)
+            sent_group = []
+            sent_tokens = 0
+            for sent in sents:
+                st = estimate_tokens(sent)
+                if sent_tokens + st > TARGET_CHUNK_TOKENS and sent_group:
+                    result.append(" ".join(sent_group))
+                    overlap = sent_group[-OVERLAP_SENTENCES:] if len(sent_group) > OVERLAP_SENTENCES else []
+                    sent_group = overlap
+                    sent_tokens = sum(estimate_tokens(s) for s in sent_group)
+                sent_group.append(sent)
+                sent_tokens += st
+            if sent_group:
+                result.append(" ".join(sent_group))
+
+        elif current_tokens + block_tokens > TARGET_CHUNK_TOKENS and current:
+            result.append("\n\n".join(current))
+            current = [block]
+            current_tokens = block_tokens
+        else:
+            current.append(block)
+            current_tokens += block_tokens
+
+    if current:
+        result.append("\n\n".join(current))
+
+    return result
+
+
+def chunk_code_table(text):
+    lines = text.split("\n")
+    chunks = []
+    current_lines = []
+    current_tokens = 0
+
+    header_line = None
+    for line in lines:
+        stripped = line.strip()
+        if not stripped:
+            continue
+
+        if re.match(r"^(?:CPT|HCPCS|Diagnosis|ICD-10)\s+(?:Code|Description)", stripped, re.IGNORECASE):
+            header_line = stripped
+            continue
+
+        if re.match(r"^The following list\(s\)", stripped):
+            continue
+        if re.match(r"^CPT®?\s+is a registered", stripped):
+            continue
+        if re.match(r"^Listing of a code", stripped):
+            continue
+
+        line_tokens = estimate_tokens(stripped)
+
+        if current_tokens + line_tokens > TARGET_CHUNK_TOKENS and current_lines:
+            chunk_text = "\n".join(current_lines)
+            if header_line:
+                chunk_text = header_line + "\n" + chunk_text
+            chunks.append(chunk_text)
+            current_lines = []
+            current_tokens = 0
+
+        current_lines.append(stripped)
+        current_tokens += line_tokens
+
+    if current_lines:
+        chunk_text = "\n".join(current_lines)
+        if header_line:
+            chunk_text = header_line + "\n" + chunk_text
+        chunks.append(chunk_text)
+
+    return chunks
+
+
+def chunk_clinical_evidence(text):
+    study_splits = re.split(
+        r'\n\n(?=(?:[A-Z][a-z]+(?:\s+(?:et al\.|and|&))?.*?\(\d{4}\))|'
+        r'(?:A\s+(?:phase|prospective|retrospective|randomized|multicenter|systematic|meta-analysis|Cochrane))|'
+        r'(?:Professional Societies|American|European|National|International))',
+        text
+    )
+
+    if len(study_splits) <= 1:
+        study_splits = text.split("\n\n")
+
+    chunks = []
+    current = []
+    current_tokens = 0
+
+    for block in study_splits:
+        block = block.strip()
+        if not block:
+            continue
+
+        block_tokens = estimate_tokens(block)
+
+        if block_tokens > MAX_CHUNK_TOKENS:
+            if current:
+                chunks.append("\n\n".join(current))
+                current = []
+                current_tokens = 0
+
+            sents = split_sentences(block)
+            sent_group = []
+            sent_tokens = 0
+            for sent in sents:
+                st = estimate_tokens(sent)
+                if sent_tokens + st > TARGET_CHUNK_TOKENS and sent_group:
+                    chunks.append(" ".join(sent_group))
+                    overlap = sent_group[-OVERLAP_SENTENCES:] if len(sent_group) > OVERLAP_SENTENCES else []
+                    sent_group = overlap
+                    sent_tokens = sum(estimate_tokens(s) for s in sent_group)
+                sent_group.append(sent)
+                sent_tokens += st
+            if sent_group:
+                chunks.append(" ".join(sent_group))
+
+        elif current_tokens + block_tokens > MAX_CHUNK_TOKENS and current:
+            chunks.append("\n\n".join(current))
+            current = [block]
+            current_tokens = block_tokens
+        else:
+            current.append(block)
+            current_tokens += block_tokens
+
+    if current:
+        chunks.append("\n\n".join(current))
+
+    return chunks
+
+
+def chunk_section(section_name, content, policy_name):
+    if section_name in BOILERPLATE_SECTIONS:
+        return []
+
+    if section_name == "Applicable Codes" or section_name == "Coverage Summary":
+        return chunk_code_table(content)
+
+    if section_name == "Clinical Evidence":
+        return chunk_clinical_evidence(content)
+
+    if section_name in ("Coverage Rationale", "Application", "Definitions",
+                        "Documentation Requirements", "Medical Records Documentation Used for Reviews"):
+        return chunk_by_criteria(content, policy_name)
+
+    tokens = estimate_tokens(content)
+    if tokens <= TARGET_CHUNK_TOKENS:
+        return [content]
+
+    return chunk_by_criteria(content, policy_name)
+
+
+def main():
+    with open(INPUT_FILE, "r", encoding="utf-8") as f:
+        policies = json.load(f)
+
+    all_chunks = []
+
+    for policy in tqdm(policies, desc="Creating chunks"):
+        policy_name = policy["policy_name"]
+        policy_number = policy.get("policy_number", "")
+        effective_date = policy.get("effective_date", "")
+        plan_type = policy.get("plan_type", "")
+        doc_type = policy.get("doc_type", "")
+
+        for section_data in policy["sections"]:
+            section_name = section_data["section"]
+            content = section_data["content"]
+            page_start = section_data.get("page_start", 0)
+            page_end = section_data.get("page_end", 0)
+
+            if section_name in BOILERPLATE_SECTIONS:
+                continue
+
+            text_chunks = chunk_section(section_name, content, policy_name)
+
+            for idx, chunk_text in enumerate(text_chunks):
+                chunk_text = chunk_text.strip()
+                if not chunk_text or len(chunk_text) < 20:
+                    continue
+
+                all_chunks.append({
+                    "id": chunk_id(policy_name, section_name, idx),
+                    "policy_name": policy_name,
+                    "policy_number": policy_number,
+                    "effective_date": effective_date,
+                    "plan_type": plan_type,
+                    "doc_type": doc_type,
+                    "section": section_name,
+                    "page_start": page_start,
+                    "page_end": page_end,
+                    "chunk_index": idx,
+                    "total_chunks_in_section": len(text_chunks),
+                    "text": chunk_text,
+                })
+
+    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
+        json.dump(all_chunks, f, indent=2, ensure_ascii=False)
+
+    print(f"Total chunks: {len(all_chunks)}")
+    print(f"Policies processed: {len(policies)}")
+    print(f"Saved to: {OUTPUT_FILE}")
+
+    section_counts = {}
+    for c in all_chunks:
+        section_counts[c["section"]] = section_counts.get(c["section"], 0) + 1
+    print("\nChunks per section:")
+    for sec, count in sorted(section_counts.items(), key=lambda x: -x[1]):
+        print(f"  {sec}: {count}")
+
+
+if __name__ == "__main__":
+    main()

scraper/download_policies.py

@@ -0,0 +1,67 @@
+import os
+import requests
+from bs4 import BeautifulSoup
+from urllib.parse import urljoin
+from tqdm import tqdm
+
+BASE_URL = "https://www.uhcprovider.com"
+PAGE_URL = "https://www.uhcprovider.com/en/policies-protocols/commercial-policies/commercial-medical-drug-policies.html"
+
+SAVE_DIR = "data/pdfs"
+os.makedirs(SAVE_DIR, exist_ok=True)
+
+
+def get_pdf_links():
+
+    print("Fetching webpage...")
+
+    response = requests.get(PAGE_URL)
+    soup = BeautifulSoup(response.text, "html.parser")
+
+    pdf_links = []
+
+    for link in soup.find_all("a", href=True):
+
+        href = link["href"]
+
+        if ".pdf" in href.lower():
+            full_link = urljoin(BASE_URL, href)
+            name = link.text.strip()
+
+            pdf_links.append({
+                "name": name,
+                "url": full_link
+            })
+
+    print(f"Found {len(pdf_links)} PDF policies")
+
+    return pdf_links
+
+
+def download_pdfs(pdf_links):
+
+    for pdf in tqdm(pdf_links):
+
+        filename = pdf["url"].split("/")[-1]
+        path = os.path.join(SAVE_DIR, filename)
+
+        if os.path.exists(path):
+            continue
+
+        try:
+            r = requests.get(pdf["url"], timeout=60)
+
+            with open(path, "wb") as f:
+                f.write(r.content)
+
+        except Exception as e:
+            print("Failed:", pdf["url"], e)
+
+
+if __name__ == "__main__":
+
+    links = get_pdf_links()
+
+    download_pdfs(links)
+
+    print("Download complete")

scraper/extract_pdf_text.py

@@ -0,0 +1,403 @@
+import os
+import re
+import json
+import pdfplumber
+from tqdm import tqdm
+from bs4 import BeautifulSoup
+from dataclasses import dataclass, field, asdict
+from typing import Optional
+
+PDF_DIR = "data/pdfs"
+OUTPUT_FILE = "data/processed/extracted_sections.json"
+
+SKIP_FILES = {
+    "TOU-UHCPROVIDER-COM-EN.pdf",
+    "OSPP-UHCPROVIDER-COM-EN.pdf",
+}
+
+SECTION_HEADERS = [
+    "Instructions for Use",
+    "Coverage Rationale",
+    "Coverage Summary",
+    "Application",
+    "Medical Records Documentation Used for Reviews",
+    "Documentation Requirements",
+    "Definitions",
+    "Applicable Codes",
+    "Description of Services",
+    "Benefit Considerations",
+    "Clinical Evidence",
+    "Background",
+    "U.S. Food and Drug Administration",
+    "Centers for Medicare and Medicaid Services",
+    "References",
+    "Policy History/Revision Information",
+    "Frequently Asked Questions",
+]
+
+SKIP_SECTIONS = {
+    "Instructions for Use",
+    "Policy History/Revision Information",
+}
+
+PAGE_HEADER_PATTERNS = [
+    re.compile(r"^.{0,120}Page\s+\d+\s+of\s+\d+\s*$"),
+    re.compile(r"^UnitedHealthcare.*(?:Medical|Drug)\s+(?:Policy|Benefit).*(?:Effective|Policy)"),
+    re.compile(r"^Proprietary Information of UnitedHealthcare"),
+    re.compile(r"^©\s*\d{4}"),
+    re.compile(r"^Effective\s+\d{2}/\d{2}/\d{4}\s*$"),
+]
+
+SIDEBAR_PATTERNS = [
+    re.compile(r"^(?:Related\s+)?(?:Commercial|Community\s+Plan|Medicare\s+Advantage)\s+(?:Policy|Policies)", re.IGNORECASE),
+    re.compile(r"^Related\s+(?:Commercial|List)", re.IGNORECASE),
+    re.compile(r"^Medicare\s+Advantage\s+Policy", re.IGNORECASE),
+]
+
+POLICY_NUMBER_RE = re.compile(r"Policy\s+Number:\s*(\S+)")
+EFFECTIVE_DATE_RE = re.compile(r"Effective\s+Date:\s*(.+?)(?:\s{2,}|$)")
+PLAN_TYPE_RE = re.compile(r"UnitedHealthcare®?\s+(Commercial.*?)$", re.MULTILINE)
+
+
+@dataclass
+class PolicySection:
+    section: str
+    content: str
+    page_start: int
+    page_end: int
+
+
+@dataclass
+class PolicyDocument:
+    filename: str
+    policy_name: str
+    policy_number: str
+    effective_date: str
+    plan_type: str
+    doc_type: str
+    sections: list = field(default_factory=list)
+
+
+def is_html_file(path):
+    try:
+        with open(path, "rb") as f:
+            start = f.read(200).decode(errors="ignore").lower()
+            return "<html" in start or "<!doctype html" in start
+    except Exception:
+        return False
+
+
+def is_page_header(line):
+    stripped = line.strip()
+    if not stripped:
+        return True
+    for pat in PAGE_HEADER_PATTERNS:
+        if pat.search(stripped):
+            return True
+    return False
+
+
+def is_toc_line(line):
+    stripped = line.strip()
+    if re.match(r"^Table of Contents\s*Page?\s*$", stripped, re.IGNORECASE):
+        return True
+    if re.match(r"^.{3,80}\s*\.{3,}\s*\d+\s*$", stripped):
+        return True
+    return False
+
+
+def is_sidebar_start(line):
+    stripped = line.strip()
+    for pat in SIDEBAR_PATTERNS:
+        if pat.match(stripped):
+            return True
+    return False
+
+
+def detect_section(line):
+    stripped = line.strip()
+    for header in SECTION_HEADERS:
+        if stripped == header or stripped.startswith(header + "\n"):
+            return header
+        if re.match(re.escape(header) + r"\s*$", stripped):
+            return header
+    return None
+
+
+def extract_metadata(full_text, filename):
+    policy_name = os.path.basename(filename).replace(".pdf", "")
+
+    policy_number = ""
+    m = POLICY_NUMBER_RE.search(full_text[:2000])
+    if m:
+        policy_number = m.group(1).strip()
+
+    effective_date = ""
+    m = EFFECTIVE_DATE_RE.search(full_text[:2000])
+    if m:
+        effective_date = m.group(1).strip()
+
+    plan_type = ""
+    m = PLAN_TYPE_RE.search(full_text[:1000])
+    if m:
+        plan_type = m.group(1).strip()
+
+    doc_type = "Medical Policy"
+    if "Medical Benefit Drug Policy" in full_text[:1000]:
+        doc_type = "Medical Benefit Drug Policy"
+    elif "Medical Policy Update Bulletin" in full_text[:500]:
+        doc_type = "Update Bulletin"
+
+    return policy_name, policy_number, effective_date, plan_type, doc_type
+
+
+def clean_page_text(text):
+    lines = text.split("\n")
+    cleaned = []
+    in_sidebar = False
+    in_toc = False
+
+    for line in lines:
+        if is_page_header(line):
+            continue
+
+        if is_toc_line(line):
+            in_toc = True
+            continue
+
+        if in_toc:
+            if re.match(r"^.{3,80}\s*\.{3,}\s*\d+\s*$", line.strip()):
+                continue
+            stripped = line.strip()
+            if stripped and not re.search(r"\.{3,}", stripped):
+                sec = detect_section(stripped)
+                if not sec:
+                    in_toc = False
+
+            if in_toc:
+                continue
+
+        if is_sidebar_start(line):
+            in_sidebar = True
+            continue
+
+        if in_sidebar:
+            stripped = line.strip()
+            if stripped.startswith("•") or stripped.startswith("–") or not stripped:
+                continue
+            sec = detect_section(stripped)
+            if sec or (stripped and not stripped.startswith("•")):
+                in_sidebar = False
+                if sec:
+                    cleaned.append(line)
+                    continue
+                cleaned.append(line)
+                continue
+            continue
+
+        cleaned.append(line)
+
+    return "\n".join(cleaned)
+
+
+def extract_pages_pdf(pdf_path):
+    pages = []
+    try:
+        with pdfplumber.open(pdf_path) as pdf:
+            for page_num, page in enumerate(pdf.pages, start=1):
+                text = page.extract_text()
+                if text:
+                    pages.append((page_num, text))
+
+                tables = page.extract_tables()
+                if tables:
+                    for table in tables:
+                        table_text = format_table(table)
+                        if table_text:
+                            pages.append((page_num, f"[TABLE]\n{table_text}\n[/TABLE]"))
+    except Exception as e:
+        print(f"Error extracting {pdf_path}: {e}")
+    return pages
+
+
+def format_table(table):
+    if not table or len(table) < 2:
+        return ""
+    rows = []
+    for row in table:
+        if row:
+            cells = [str(cell).strip() if cell else "" for cell in row]
+            if any(cells):
+                rows.append(" | ".join(cells))
+    return "\n".join(rows)
+
+
+def build_paragraphs(text):
+    lines = text.split("\n")
+    paragraphs = []
+    current = []
+
+    for line in lines:
+        stripped = line.strip()
+        if not stripped:
+            if current:
+                paragraphs.append(" ".join(current))
+                current = []
+            continue
+
+        is_bullet = bool(re.match(r"^[•\-–▪o]\s", stripped))
+        is_numbered = bool(re.match(r"^\d+[\.\)]\s", stripped))
+        is_lettered = bool(re.match(r"^[a-z][\.\)]\s", stripped))
+        is_list_item = is_bullet or is_numbered or is_lettered
+
+        if is_list_item:
+            if current:
+                paragraphs.append(" ".join(current))
+                current = []
+            current.append(stripped)
+        elif stripped.startswith("o\t") or stripped.startswith("o "):
+            if current:
+                paragraphs.append(" ".join(current))
+                current = []
+            current.append(stripped)
+        else:
+            current.append(stripped)
+
+    if current:
+        paragraphs.append(" ".join(current))
+
+    return paragraphs
+
+
+def segment_into_sections(pages):
+    all_text_by_section = []
+    current_section = ""
+    current_content = []
+    current_page_start = 1
+
+    for page_num, raw_text in pages:
+        cleaned = clean_page_text(raw_text)
+        lines = cleaned.split("\n")
+
+        for line in lines:
+            stripped = line.strip()
+            if not stripped:
+                current_content.append("")
+                continue
+
+            sec = detect_section(stripped)
+            if sec:
+                if current_content:
+                    text = "\n".join(current_content).strip()
+                    if text and current_section:
+                        all_text_by_section.append(PolicySection(
+                            section=current_section,
+                            content=text,
+                            page_start=current_page_start,
+                            page_end=page_num
+                        ))
+                current_section = sec
+                current_content = []
+                current_page_start = page_num
+                continue
+
+            current_content.append(stripped)
+
+    if current_content and current_section:
+        text = "\n".join(current_content).strip()
+        if text:
+            last_page = pages[-1][0] if pages else 1
+            all_text_by_section.append(PolicySection(
+                section=current_section,
+                content=text,
+                page_start=current_page_start,
+                page_end=last_page
+            ))
+
+    return all_text_by_section
+
+
+def extract_html(path, filename):
+    with open(path, "r", encoding="utf-8", errors="ignore") as f:
+        soup = BeautifulSoup(f, "html.parser")
+    text = soup.get_text("\n")
+
+    policy_name, policy_number, effective_date, plan_type, doc_type = extract_metadata(text, filename)
+
+    pages = [(1, text)]
+    sections = segment_into_sections(pages)
+
+    return PolicyDocument(
+        filename=filename,
+        policy_name=policy_name,
+        policy_number=policy_number,
+        effective_date=effective_date,
+        plan_type=plan_type,
+        doc_type=doc_type,
+        sections=[asdict(s) for s in sections if s.section not in SKIP_SECTIONS]
+    )
+
+
+def extract_policy(pdf_path, filename):
+    pages = extract_pages_pdf(pdf_path)
+
+    if not pages:
+        return None
+
+    full_text = "\n".join(text for _, text in pages[:3])
+    policy_name, policy_number, effective_date, plan_type, doc_type = extract_metadata(full_text, filename)
+
+    sections = segment_into_sections(pages)
+
+    filtered_sections = []
+    for sec in sections:
+        if sec.section in SKIP_SECTIONS:
+            continue
+
+        paragraphs = build_paragraphs(sec.content)
+        cleaned_content = "\n\n".join(p for p in paragraphs if len(p.strip()) > 10)
+
+        if cleaned_content.strip():
+            sec.content = cleaned_content
+            filtered_sections.append(sec)
+
+    return PolicyDocument(
+        filename=filename,
+        policy_name=policy_name,
+        policy_number=policy_number,
+        effective_date=effective_date,
+        plan_type=plan_type,
+        doc_type=doc_type,
+        sections=[asdict(s) for s in filtered_sections]
+    )
+
+
+def main():
+    all_policies = []
+
+    pdfs = [f for f in os.listdir(PDF_DIR) if f not in SKIP_FILES]
+    pdfs.sort()
+
+    for filename in tqdm(pdfs, desc="Extracting policies"):
+        path = os.path.join(PDF_DIR, filename)
+
+        if is_html_file(path):
+            doc = extract_html(path, filename)
+        else:
+            doc = extract_policy(path, filename)
+
+        if doc and doc.sections:
+            all_policies.append(asdict(doc))
+
+    os.makedirs(os.path.dirname(OUTPUT_FILE), exist_ok=True)
+
+    with open(OUTPUT_FILE, "w", encoding="utf-8") as f:
+        json.dump(all_policies, f, indent=2, ensure_ascii=False)
+
+    total_sections = sum(len(p["sections"]) for p in all_policies)
+    print(f"Extracted {len(all_policies)} policies with {total_sections} sections")
+    print(f"Saved to: {OUTPUT_FILE}")
+
+
+if __name__ == "__main__":
+    main()