docs/interview-talking-points.md

# Interview Talking Points

Rehearsed answers mapped to likely questions. Each answer is anchored in a
concrete decision you made in this codebase, so you can always point at code
when pressed for specifics.

## "Walk me through your RAG pipeline."

Ingestion is a three-step transform. I pull MTSamples psychiatry notes from
a CSV, chunk them with a section-aware regex that recognizes clinical headers
like HPI, ASSESSMENT, and PLAN — with a recursive character splitter as a
fallback for notes that don't follow the template. Each chunk gets embedded
with S-PubMedBert-MS-MARCO, a BERT-family encoder pretrained on biomedical
text, producing a 768-dimensional vector. Chunks, metadata, and vectors land
in a Postgres table with an HNSW index for approximate nearest neighbor
search and a GIN index on a tsvector column for BM25.

At query time, I embed the user query with the same model, run hybrid search
— cosine similarity plus BM25, weighted 70/30 — and take the top 5 chunks.
Those get formatted into a prompt with a strict system instruction: only use
the provided context, cite every claim with `[chunk_id]`, and refuse with a
specific string if the answer isn't in the context. After generation, I
validate that every cited chunk_id actually appears in the retrieved set —
if one doesn't, the model hallucinated a citation, and we flag the response.

FastAPI serves this via a `/query` endpoint with Pydantic validation and
audit-style logging, Streamlit demos it with a side-by-side view of the
answer and retrieved chunks, and Docker Compose ties it all together.

## "How do you prevent hallucinations?"

Three layers of defense.

Layer one is retrieval quality. Better chunks mean less hallucination
pressure on the LLM. I use a domain-specific embedder, hybrid search to
catch both semantic matches and exact strings, and I evaluate precision@5
on a hand-labeled set so I know when retrieval is failing.

Layer two is prompt engineering. The system prompt forces citations for
every claim and specifies an exact refusal string when the context doesn't
support an answer. Exact-string refusal is deliberate — soft refusals like
"I'm not sure but..." still hallucinate. An exact string is machine-checkable.

Layer three is post-generation validation. I parse the citations out of the
response and verify each cited chunk_id appears in the retrieved set. An
invalid citation is a strong signal the model fabricated content. In
production I'd extend this to faithfulness scoring — checking whether each
claim is actually supported by its cited chunk, probably via a cross-encoder
or a second LLM call.

## "Why pgvector over Pinecone or Weaviate?"

For a dataset that fits on a laptop, pgvector wins on operational simplicity
— one image, one query language, metadata filtering and vector search in the
same SQL statement. Transactional guarantees come free. At multi-million-chunk
scale, the tradeoffs shift and a managed service might win on operational
overhead. But for a prototype, pulling in a managed vector DB is
over-engineering, and it signals "I follow tutorials" rather than "I make
considered choices."

## "How do you handle PHI and HIPAA?"

This project uses MTSamples only — public, de-identified data — so HIPAA
doesn't technically apply. But I built it as if it did, because those habits
need to exist before real data shows up.

Specifically: API keys live only in a gitignored `.env` file, never in code.
Audit logs contain query hashes, not raw query text — because in production
queries might reference patient identifiers. All SQL is parameterized, so
query content can't be injected. Containers run as non-root. CORS is locked
down to the UI origin only. The health endpoint reveals nothing about
internals.

For a real deployment with PHI: BAA with Anthropic first, or swap the
generator for a self-hosted Llama via vLLM in a VPC. Add de-identification
in ingestion — NER for names, dates, MRNs. Encrypt the vector DB volume at
rest. Immutable audit logging to a WORM bucket. Role-based access controls
with short-lived credentials. Network segmentation with an egress proxy
logging every outbound API call.

## "Why didn't you fine-tune the embedder?"

Two reasons. First, without a labeled evaluation set big enough to measure
a real delta, fine-tuning is theatrical — you produce a model and a feeling,
not a measurable improvement. Second, the base model is already
transfer-learned twice: BERT → PubMedBERT for biomedical domain,
PubMedBERT → S-PubMedBERT-MS-MARCO for the sentence-embedding objective.
It's domain-appropriate out of the box.

That said, here's exactly what I'd do next if I had two more days. Generate
synthetic query-chunk pairs with Claude — maybe 1000-2000 triplets of
(query, relevant_chunk, irrelevant_chunk). Have a clinician validate a
random sample. Fine-tune with MultipleNegativesRankingLoss. Measure
precision@5 and MRR against the current baseline before shipping. If the
delta is under 3 points, keep the off-the-shelf model and spend the effort
elsewhere. That's the production mindset — measure, then decide.

## "What's the difference between a BERT embedder and a GPT-style LLM?"

Different architectures for different jobs. BERT is a bi-directional encoder
— every token attends to every other token in both directions, and the model
produces a single fixed-size vector per input. Trained with masked language
modeling and then contrastive objectives for embedders. It's good at "what
does this text mean?" in a single vector.

GPT-style models are autoregressive decoders — left-to-right, next-token
prediction. They're good at generation, not at producing single
representations. You can extract embeddings from them, but they're generally
worse than purpose-built bi-directional encoders for retrieval.

In this pipeline, the BERT-family encoder handles retrieval, the Claude
decoder handles generation. Right tool for each job.

## "How would you scale this to millions of notes?"

Two levers. Ingestion becomes embarrassingly parallel — PySpark or Ray
across a cluster, embedding in batches on GPU nodes. I'd keep pgvector for
moderate scale (tens of millions of chunks) with proper HNSW tuning and
partitioning, or move to a managed service like Databricks Vector Search
beyond that.

At query time, the bottleneck moves to two places: the ANN index (tune
`ef_search` for the precision/latency tradeoff), and the LLM call (add a
reranker so you can retrieve top-50 cheaply and reranker-narrow to top-5
before paying the generation cost). Prompt caching on the system prompt
saves real money at scale.

Monitoring becomes essential. Track retrieval precision, faithfulness,
latency percentiles, refusal rate. Drift in any of these is the first
signal that the index or the corpus has changed in ways that hurt quality.

## "How do you work with clinical stakeholders?"

Demo-driven. A Streamlit prototype in a meeting is worth ten slides.
Clinicians tell you almost instantly what's wrong with retrieval when they
can see both the answer and the sources side-by-side — which is why my UI
shows retrieved chunks alongside every answer. Transparency isn't a
nice-to-have in healthcare; it's the feature that makes clinicians trust
the tool enough to use it.

Iteration cycles: build minimum viable retrieval, show it, collect specific
failure cases from the clinicians, bake those into the eval set, improve
against the eval, show the delta. Repeat. Every "the system got this wrong"
becomes a test case that prevents regression.

## "What would you do differently next time?"

Three things, roughly in order.

Build the eval harness first. I built the pipeline and then the eval, which
meant I had to retrofit test queries around what the system already did.
Writing the eval first would have forced clearer thinking about what
"success" meant.

Invest earlier in reranking. Cross-encoder reranking on top-20 candidates
improves precision significantly for marginal latency cost. I left it as a
"next step" but it should have been in V1.

Make evaluation continuous. Right now `run_eval.py` is a one-shot script.
In production I'd want these metrics tracked over time, alerting on
regression, with a versioned eval set that grows as clinicians flag new
failure cases.

## Bridge lines for your existing experience

- **Data pipelines:** "I've built ETL pipelines in Python — ingestion here is
  the same pattern, with a neural network doing the transform step."
- **PostgreSQL:** "I know Postgres well, so pgvector let me reuse all my
  existing indexing and query optimization intuition."
- **FastAPI:** "This is my normal API stack — the only difference is the
  endpoint returns both the answer and the sources for auditability."
- **Docker:** "Standard multi-container setup. The interesting constraint
  was container hardening — non-root users, no secrets baked in."
- **Streamlit:** "I use Streamlit for demos that need to land with
  non-technical stakeholders. Here it's showing clinical users both the
  answer and the evidence."