# Design Notes ## Key decisions and tradeoffs ### API target: own implementation Instead of wrapping a third-party fake API, the client wraps this project's own FastAPI backend. This means the client and the API are co-designed — the typed models on both sides stay in sync by design. The tradeoff: less realistic than wrapping an external API you don't control, but the test surface is richer and the integration tests verify real business logic, not just HTTP plumbing. ### Two-layer evaluation (L1 live / L2 batch) L1 runs on every query inline (~1-2s overhead). L2 runs offline against a golden dataset. The split is a deliberate latency/depth tradeoff: LLM-judged metrics (contextual precision, reverse-question relevancy) add 30+ seconds per pair — unacceptable live, fine in batch. The golden dataset is the contract; L2 is the regression gate. ### Deterministic chain_terminology over LLM judge The terminology check is a dict lookup, not a model call. Zero latency, zero cost, zero false negatives on known mappings. The tradeoff: it only catches terms in the catalog — novel terminology drift goes undetected. An LLM judge would catch drift but would introduce latency and non-determinism into a metric that must be auditable. ### In-memory retrieval over vector database KB size is 8-9 docs per domain. Encoding them at startup and doing cosine search at query time adds ~2ms retrieval overhead with no infrastructure dependency. A vector DB (Chroma, pgvector) would add operational complexity with zero retrieval quality gain at this scale. ### httpx + tenacity for the client `httpx` is the modern alternative to `requests`: native async support if needed later, cleaner timeout API, better type annotations. `tenacity` separates retry policy from request logic cleanly — the retry decorator is readable and testable independently from the HTTP code. ### Integration tests are read-only by design The API has no mutable state: queries don't persist, no records are created or deleted. Cleanup is therefore trivially satisfied — there is nothing to clean up. This is called out explicitly because it's a deliberate architectural choice, not an oversight. A stateful API (task creation, deletion) would require explicit teardown fixtures. --- ## NLI model selection — what was tried and why The faithfulness grader went through three models before converging: **Vectara HHEM v2** (`vectara/hallucination_evaluation_model`) — purpose-built for RAG faithfulness, not general NLI. The correct model for this task. Unusable: the checkpoint is missing `t5.transformer.encoder.embed_tokens.weight`. The embedding matrix is zero-initialized (`std=0.0`), producing constant 0.502 probability for every input. Diagnosed via weight inspection, not error message. **`cross-encoder/nli-deberta-v3-small`** (first attempt, paragraph-level) — 3-class NLI (contradiction / entailment / neutral). Correct model family, wrong input format. NLI cross-encoders are trained on sentence-pair inputs (SNLI/MNLI). Feeding a 3–4 sentence KB paragraph as the premise causes entailment scores to collapse — verbatim text scores `ent=0.002`, treated as neutral. Root cause: model distributes probability across longer sequences in ways not seen during training. **`cross-encoder/nli-deberta-v3-small` (sentence-level)** — same model, fixed by splitting KB chunks into individual sentences before scoring. Verbatim: `ent=0.995`. Aliased terms ("item registry" vs "product catalog (item registry)"): `ent=0.989`. Hallucinated facts: `ent≈0.000`, contradiction≈1.0. This is the current implementation. **Key insight:** the NLI model selection problem is a data format problem as much as a model selection problem. The same model produces correct results at sentence level and degenerate results at paragraph level. --- ## Alternative judge approaches considered ### Ollama (local LLM judge) Ollama can run Llama 3 / Mistral locally, making it a zero-cost alternative to HF Inference API for both generation and LLM-as-judge evaluation. Tradeoffs: requires local GPU or accepts slower CPU inference; no external API rate limits; outputs are fully reproducible since the model version is pinned. For the faithfulness judge specifically, a local `llama3` via Ollama would remove the dependency on HF token entirely and allow offline eval runs. ### Prometheus (LLM eval framework) [Prometheus-2](https://huggingface.co/prometheus-eval/prometheus-7b-v2.0) is a 7B model fine-tuned specifically for evaluation tasks — outputs a score + rationale in a structured format designed for rubric-based grading. It's a drop-in replacement for GPT-4/Claude as eval judge, runs via Ollama or HF Inference, and is purpose-built for the kind of faithfulness + relevancy scoring done in `eval/metrics.py`. The tradeoff vs. the current sentence-level NLI approach: Prometheus is slower (7B vs purpose-built cross-encoder) but produces a human-readable rationale alongside the score, which is more interpretable for audit and debugging. **Why not used here:** the cross-encoder NLI approach runs faster and requires no prompt engineering. Prometheus would be the right choice if rationale logging is a compliance requirement. --- ## What another 4 hours would add - **`eval/metrics.py` — L2 LLM metrics**: contextual precision (chunk ranking), contextual recall (coverage), and answer correctness against full reference answers. Currently only keyphrase coverage is used as a proxy. - **Async client**: `httpx.AsyncClient` variant for high-concurrency load testing. - **Property-based tests**: `hypothesis` to fuzz `check_terminology` and graders with generated strings — catches edge cases the golden dataset doesn't cover. - **CI pipeline**: GitHub Actions running `make lint`, `make type-check`, `make test` on every PR. Integration tests gated on a self-hosted runner with the API running. - **Threshold calibration report**: `eval/calibrate.py` exists and runs graders against golden-dataset expected answers — threshold calibration is now a single command, not a missing feature. Actual threshold adjustments require reviewing the output against real query distributions. ## Gate 5 audit gaps addressed - **Faithfulness false negatives on refusals**: `_is_refusal()` detects "I don't have enough information" responses and returns score=1.0 — no factual claims, trivially faithful. - **Partial grounding blind spot**: faithfulness now uses claim-level decomposition (`grade_faithfulness_decomposed`). Response split into sentences; each verified independently. Score = supported_claims / total_claims. A response with one hallucinated sentence in three now scores 0.667, not 1.0. - **No escalation path**: `overall_pass=False` now emits a structured `EVAL_FAIL` WARNING log entry and sets `flagged: true` in the response payload. UI shows a red banner. - **Cold-start latency**: embedder and NLI model pre-warmed at startup in the FastAPI lifespan. - **Happy-path-only golden dataset**: 4 adversarial pairs added (vague query, rival-term prompt injection, multi-doc synthesis, hallucination bait). - **No drift detection**: added `eval/drift.py` — KS two-sample test per metric, compares live telemetry scores against golden-dataset baseline. Detects faithfulness degradation at p < 0.05 with ~40% traffic degradation across 40+ events. --- ## Where LLM assistance helped and where it misled **Helped:** - Scaffolding the full project structure (backend, client, tests, config) in a single session without losing consistency across files. - Writing the faithfulness prompt in a way that reliably returns structured JSON — the few-shot JSON format in the prompt was a suggested pattern that works. - Catching that `except Exception` in the faithfulness grader was too broad and replacing it with `(json.JSONDecodeError, anthropic.APIError)`. - Identifying that `_build_index_by_domain` was defined twice in pipeline.py (duplicate introduced during an edit session) — caught during code review. **Misled or required correction:** - Initially used `lru_cache` on a function that takes a `SentenceTransformer` instance as an argument — unhashable, so the cache silently failed. Required switching to a module-level dict cache. - Generated a dead loop in `rosetta.py` (iterating over terms with `continue` but no code after the continue branch) that did nothing. The logic existed in a comment describing intent but was never implemented. Caught in review. - Suggested a fictional client name that conflicted with a real company. Required renaming before the repo went public.