Spaces:
Sleeping
docs: stale-references sweep — align legacy docs with ADR-044 + 8-gate state
Browse files11 files fixed per 2026-05-27 docs-audit (P0 / P1 / P2 / one P3):
P0 — Upload pipeline most-likely-to-mislead:
- 70-docs/20-data-pipeline/ingestion-policy.md: wholesale §"Exception — user uploads" rewrite; dual-write table row; date bumped + ADR-044 cross-ref
- kb/security/INDEX.md: "5 gates" → "8 gates" with full table extension
P1 — Status inversions:
- 70-docs/60-decisions/ADR-003-curated-corpus.md: "Locked" banner now says "partially superseded by ADR-044"; counts 190/19 → 148/21; quarantine-only framing flipped to dual-write
- 70-docs/60-decisions/README.md: ADR-003 status row updated
- 70-docs/10-architecture/safety-architecture.md: F-13 "Not applicable — D-003" → "LIVE per ADR-044, mitigated by 8 gates" (resolved internal contradiction)
- 70-docs/00-overview/problem-statement.md: §5.4 qualified as v1 intent + ADR-044 update; §7 non-goal bullet replaced with cross-session-retrieval caveat
P2 — Architecture summaries omitted dual-write:
- ARCHITECTURE.md: explicit ADR-044 dual-write paragraph; count 150/20 → 148/21
- 70-docs/10-architecture/system-overview.md: retrieval bullet extended with dual-write language
- kb/AUDIT_TRAIL.md: D-003 artifact row expanded to include backend/uploaded_docs.py + security.py
P3 — History-of-record + audit appendix:
- 70-docs/60-decisions/legacy-decisions-monolith.md: D-003 gets one-line ADR-044 supersession note
- 80-audit/ENTERPRISE_AUDIT.md: new "Sprint 2026-05-27 (KI-225..KI-333)" addendum after the 05-15 sprint; covers single-brain consolidation + upload-pipeline rebuild + 4 hardening KIs
DO NOT TOUCH list honored: README.md, CLAUDE.md, ADR-044, catalogued-corpus manifests (corpus_urls.md, information_source_map.json), 40-data/policy_facts/*.json all untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 70-docs/00-overview/problem-statement.md +2 -2
- 70-docs/10-architecture/safety-architecture.md +1 -1
- 70-docs/10-architecture/system-overview.md +4 -3
- 70-docs/20-data-pipeline/ingestion-policy.md +54 -27
- 70-docs/60-decisions/ADR-003-curated-corpus.md +4 -4
- 70-docs/60-decisions/ADR-044-uploaded-pdf-parity.md +41 -0
- 70-docs/60-decisions/README.md +1 -1
- 70-docs/60-decisions/legacy-decisions-monolith.md +1 -0
- 80-audit/ENTERPRISE_AUDIT.md +7 -1
- ARCHITECTURE.md +6 -3
- kb/AUDIT_TRAIL.md +1 -1
- kb/security/INDEX.md +16 -8
|
@@ -146,7 +146,7 @@ Hindi and Hinglish are designed-in from day one, not retrofitted. The advisor op
|
|
| 146 |
|
| 147 |
### 5.4 Curated corpus is the product's moat
|
| 148 |
|
| 149 |
-
The user does not upload PDFs
|
| 150 |
|
| 151 |
### 5.5 Architecture is built for category expansion from day one
|
| 152 |
|
|
@@ -189,7 +189,7 @@ To protect against scope creep — and to give the interviewer something to grad
|
|
| 189 |
- **Real-time quote pulls from insurer APIs.** Out of scope; possibly out of reach commercially. We use aggregated public data and label pricing as illustrative.
|
| 190 |
- **Categories beyond Health.** Life, Motor, Travel, Cyber, etc. are explicitly v2. The architecture accommodates them; we do not populate them.
|
| 191 |
- **Multi-user / multi-tenant deployment.** v1 is single-user demo. Stateless services + canonical DB make v2 multi-tenant tractable.
|
| 192 |
-
- **
|
| 193 |
- **Mobile-native deployment.** v1 is desktop browser. Mobile responsive is a stretch goal.
|
| 194 |
- **Real-time co-browsing with a human advisor.** v2 handoff feature.
|
| 195 |
- **Persistent user accounts across sessions.** v1 is session-scoped.
|
|
|
|
| 146 |
|
| 147 |
### 5.4 Curated corpus is the product's moat
|
| 148 |
|
| 149 |
+
**v1 design intent:** The user does not upload PDFs (we pre-acquire, normalise, and structurally extract from the curated corpus so quality is under our control). **As of 2026-05-27 (ADR-044):** uploads are now first-class via an 8-gate defence + a deterministic heuristic floor + the same Gemini extraction chain the catalogued 148 use offline. The curated-corpus quality moat is preserved; uploads add a per-user surface area we control via the gates + the floor. The corpus is the product's defensibility relative to a generic RAG-over-anything bot.
|
| 150 |
|
| 151 |
### 5.5 Architecture is built for category expansion from day one
|
| 152 |
|
|
|
|
| 189 |
- **Real-time quote pulls from insurer APIs.** Out of scope; possibly out of reach commercially. We use aggregated public data and label pricing as illustrative.
|
| 190 |
- **Categories beyond Health.** Life, Motor, Travel, Cyber, etc. are explicitly v2. The architecture accommodates them; we do not populate them.
|
| 191 |
- **Multi-user / multi-tenant deployment.** v1 is single-user demo. Stateless services + canonical DB make v2 multi-tenant tractable.
|
| 192 |
+
- **Cross-session retrieval scoping for uploaded PDFs.** Live uploads dual-write into the global `policies` Chroma collection per ADR-044 (2026-05-27); full per-session scoping is a tracked follow-up. (See §5.4.)
|
| 193 |
- **Mobile-native deployment.** v1 is desktop browser. Mobile responsive is a stretch goal.
|
| 194 |
- **Real-time co-browsing with a human advisor.** v2 handoff feature.
|
| 195 |
- **Persistent user accounts across sessions.** v1 is session-scoped.
|
|
@@ -149,7 +149,7 @@ In a regulated BFSI domain, the worst failure isn't "the bot looks slow" — it'
|
|
| 149 |
|
| 150 |
### F-13 — User uploads a malicious PDF (XSS / macro)
|
| 151 |
|
| 152 |
-
**Description:**
|
| 153 |
|
| 154 |
### F-14 — User asks for medical advice ("should I get this surgery?")
|
| 155 |
|
|
|
|
| 149 |
|
| 150 |
### F-13 — User uploads a malicious PDF (XSS / macro)
|
| 151 |
|
| 152 |
+
**Description:** F-13 is LIVE per ADR-044 (2026-05-27). Mitigated by the 8 gates in `backend/security.py` (file mechanics / dangerous PDF features / content quality / prompt injection / per-session rate limit / per-IP rate limit / encrypted-PDF reject / page-ceiling / hash dedupe). Failed uploads are logged to `logs/upload_blocks.jsonl`. The 8-gate state is the same one this file's top banner already lists.
|
| 153 |
|
| 154 |
### F-14 — User asks for medical advice ("should I get this surgery?")
|
| 155 |
|
|
@@ -37,9 +37,10 @@ authoritative architecture description:
|
|
| 37 |
transient Gemini error → small `backend/nim_fallback.py` (NVIDIA NIM) so
|
| 38 |
the turn completes. Fail-loud.
|
| 39 |
- Retrieval: **structured + vector over Chroma + BGE-small local 384-d**
|
| 40 |
-
(`rag/retrieve.py`) with a profile-tuned scorecard; shared policies
|
| 41 |
-
collection + per-session 24h quarantine for user-uploaded PDFs (
|
| 42 |
-
|
|
|
|
| 43 |
- **Next.js 16 / FastAPI**, deployed as an HF Space (Docker, `uvicorn`,
|
| 44 |
port 7860); heavy data pulled at build from the `insurance-bot-data` HF
|
| 45 |
dataset; curated facts in `40-data/`.
|
|
|
|
| 37 |
transient Gemini error → small `backend/nim_fallback.py` (NVIDIA NIM) so
|
| 38 |
the turn completes. Fail-loud.
|
| 39 |
- Retrieval: **structured + vector over Chroma + BGE-small local 384-d**
|
| 40 |
+
(`rag/retrieve.py`) with a profile-tuned scorecard; shared `policies`
|
| 41 |
+
collection + per-session 24h quarantine for user-uploaded PDFs (per
|
| 42 |
+
ADR-044, uploads dual-write into both so they become first-class
|
| 43 |
+
marketplace cards). 8 security gates in `backend/security.py`.
|
| 44 |
- **Next.js 16 / FastAPI**, deployed as an HF Space (Docker, `uvicorn`,
|
| 45 |
port 7860); heavy data pulled at build from the `insurance-bot-data` HF
|
| 46 |
dataset; curated facts in `40-data/`.
|
|
@@ -1,10 +1,13 @@
|
|
| 1 |
-
# Ingestion Policy (2026-05-14)
|
| 2 |
|
| 3 |
**Headline:** All extraction, embedding, and Chroma writes for the curated
|
| 4 |
policy corpus run on the developer's Mac. The deployed Hugging Face Space
|
| 5 |
serves pre-built indexes only. The single exception is user-uploaded PDFs,
|
| 6 |
-
which are embedded on-demand
|
| 7 |
-
|
|
|
|
|
|
|
|
|
|
| 8 |
|
| 9 |
---
|
| 10 |
|
|
@@ -16,7 +19,8 @@ Chroma collection isolated from the main `policies` corpus.
|
|
| 16 |
| Text extraction (pdfplumber → JSON) | yes | no |
|
| 17 |
| Chunking (800-token windows, 120 overlap) | yes | no |
|
| 18 |
| Embedding (BAAI/bge-small-en-v1.5) | yes | **only quarantine** |
|
| 19 |
-
| Chroma write — `policies` collection
|
|
|
|
| 20 |
| Chroma write — `user_uploads_quarantine` | no | yes (on-demand) |
|
| 21 |
| HF dataset push (`rag/vectors/**`, `rag/extracted/**`, `rag/corpus/**`) | yes | no |
|
| 22 |
| HF Space push (code only) | yes | no |
|
|
@@ -66,34 +70,57 @@ That created two bad failure modes:
|
|
| 66 |
while it churned, then either succeeded (slow, but fine) or failed deep
|
| 67 |
inside the embedder. Operators could not tell which.
|
| 68 |
2. **Resource cost on the wrong machine.** Free-tier Spaces have a CPU
|
| 69 |
-
budget and a 1 GB image cap. Embedding
|
| 70 |
-
|
| 71 |
-
|
|
|
|
|
|
|
| 72 |
|
| 73 |
Fail-fast is better: `entrypoint.sh` now validates Chroma is readable and
|
| 74 |
populated, and exits with a loud error if not. The fix is documented at
|
| 75 |
exit time (run `rag.ingest` locally, push vectors, redeploy). Total
|
| 76 |
boot-to-serving time on the Space is now seconds, not tens of minutes.
|
| 77 |
|
| 78 |
-
## Exception — user uploads
|
| 79 |
|
| 80 |
`POST /api/upload-policy` accepts an arbitrary PDF from the public web and
|
| 81 |
-
must embed it before the chatbot can answer questions about it.
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Ingestion Policy (2026-05-27, supersedes 2026-05-14 state per ADR-044)
|
| 2 |
|
| 3 |
**Headline:** All extraction, embedding, and Chroma writes for the curated
|
| 4 |
policy corpus run on the developer's Mac. The deployed Hugging Face Space
|
| 5 |
serves pre-built indexes only. The single exception is user-uploaded PDFs,
|
| 6 |
+
which are embedded on-demand and — per ADR-044 (2026-05-27) — **dual-write**
|
| 7 |
+
into BOTH a per-session `user_uploads_quarantine` Chroma collection AND the
|
| 8 |
+
global `policies` Chroma collection, so the upload becomes a first-class
|
| 9 |
+
marketplace card with the same scorecard / premium / RAG endpoints as the
|
| 10 |
+
catalogued 148.
|
| 11 |
|
| 12 |
---
|
| 13 |
|
|
|
|
| 19 |
| Text extraction (pdfplumber → JSON) | yes | no |
|
| 20 |
| Chunking (800-token windows, 120 overlap) | yes | no |
|
| 21 |
| Embedding (BAAI/bge-small-en-v1.5) | yes | **only quarantine** |
|
| 22 |
+
| Chroma write — `policies` collection (curated 148 corpus) | yes | no |
|
| 23 |
+
| Chroma write — `policies` (per-upload chunks via dual-write) | no | yes (on-demand) |
|
| 24 |
| Chroma write — `user_uploads_quarantine` | no | yes (on-demand) |
|
| 25 |
| HF dataset push (`rag/vectors/**`, `rag/extracted/**`, `rag/corpus/**`) | yes | no |
|
| 26 |
| HF Space push (code only) | yes | no |
|
|
|
|
| 70 |
while it churned, then either succeeded (slow, but fine) or failed deep
|
| 71 |
inside the embedder. Operators could not tell which.
|
| 72 |
2. **Resource cost on the wrong machine.** Free-tier Spaces have a CPU
|
| 73 |
+
budget and a 1 GB image cap. Embedding the curated corpus (148 policies
|
| 74 |
+
across 21 insurers as of 2026-05-27; older docs referencing 190/19 or
|
| 75 |
+
256 are pre-dedup file counts) on Space CPU was slower than on a
|
| 76 |
+
developer Mac and pushed boot far past the platform's health-check
|
| 77 |
+
window.
|
| 78 |
|
| 79 |
Fail-fast is better: `entrypoint.sh` now validates Chroma is readable and
|
| 80 |
populated, and exits with a loud error if not. The fix is documented at
|
| 81 |
exit time (run `rag.ingest` locally, push vectors, redeploy). Total
|
| 82 |
boot-to-serving time on the Space is now seconds, not tens of minutes.
|
| 83 |
|
| 84 |
+
## Exception — user uploads (ADR-044, 2026-05-27)
|
| 85 |
|
| 86 |
`POST /api/upload-policy` accepts an arbitrary PDF from the public web and
|
| 87 |
+
must embed + extract it before the chatbot can answer questions about it.
|
| 88 |
+
The pipeline lives in `backend/uploaded_docs.py` + `backend/security.py`
|
| 89 |
+
and is governed by **ADR-044** (with hardening bundle KI-330 / KI-331 /
|
| 90 |
+
KI-332 / KI-333; live on `e7f799a`). See README §2.8 and
|
| 91 |
+
[ADR-044](../60-decisions/ADR-044-uploaded-pdf-parity.md) for the full
|
| 92 |
+
spec.
|
| 93 |
+
|
| 94 |
+
**Dual-write model.** Unlike the original v1 design (quarantine-only),
|
| 95 |
+
every accepted upload is written into BOTH:
|
| 96 |
+
|
| 97 |
+
- a per-session `user_uploads_quarantine` Chroma collection (session-scoped,
|
| 98 |
+
24h TTL, never persisted across Space rebuilds), AND
|
| 99 |
+
- the global `policies` Chroma collection — so the uploaded policy becomes
|
| 100 |
+
a **first-class marketplace card** with the same scorecard / premium /
|
| 101 |
+
RAG endpoints as the catalogued 148 policies across 21 insurers.
|
| 102 |
+
|
| 103 |
+
**8-gate defence in `backend/security.py`** runs before any embedding:
|
| 104 |
+
(1) file mechanics + size 5KB-25MB + `%%EOF` + no executable / JS payloads,
|
| 105 |
+
(2) content quality ≥1,500 chars + ≥3 pages + ≥1 insurance keyword,
|
| 106 |
+
(3) prompt-injection sweep, (4) per-session rate limit, (5) per-IP rate
|
| 107 |
+
limit, (6) encrypted / locked PDF reject, (7) page-count ceiling >200,
|
| 108 |
+
(8) hash dedupe + reject-cache. A UIN net-new check and PDF-text fuzzy
|
| 109 |
+
match against the catalogued 148 also run beyond the 8. Failed uploads
|
| 110 |
+
log to `logs/upload_blocks.jsonl`.
|
| 111 |
+
|
| 112 |
+
**Heuristic floor — HARD guarantee.** `build_record()` runs sub-second
|
| 113 |
+
on every accepted upload and writes `record.json` BEFORE the LLM ever
|
| 114 |
+
fires. Post-KI-332 it lifts the floor from ~47% to ~65-70% via 28+
|
| 115 |
+
regex patterns. This is the parity guarantee on the fail path
|
| 116 |
+
(KI-331 + KI-333: status ↔ scorecard card always agree).
|
| 117 |
+
|
| 118 |
+
**LLM extraction chain.** hash-cache → multi-pass (≥25K chars, 7 sections
|
| 119 |
+
in parallel) → single-pass Gemini 2.5-flash (3 jittered retries) →
|
| 120 |
+
NIM fallback → heuristic floor. Provenance surfaces on status as
|
| 121 |
+
`llm_used` + `llm_response_chars`.
|
| 122 |
+
|
| 123 |
+
**Open caveat — cross-session leakage.** The dual-write into `policies`
|
| 124 |
+
means another session's retrieval can surface a user's upload. Mitigated
|
| 125 |
+
today by a per-session quarantine boost in retrieval; full per-session
|
| 126 |
+
scoping of the `policies` collection is a tracked follow-up.
|
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# ADR-003: Curated corpus over user-uploaded PDFs
|
| 2 |
|
| 3 |
-
**Status:** Locked
|
| 4 |
**Date:** 2026-05-13
|
| 5 |
|
| 6 |
## Context
|
|
@@ -12,14 +12,14 @@ A health-insurance bot needs PDF text to ground its answers. The PDFs can come f
|
|
| 12 |
|
| 13 |
## Decision
|
| 14 |
|
| 15 |
-
**Curated corpus** of
|
| 16 |
|
| 17 |
## Alternatives considered
|
| 18 |
|
| 19 |
| Approach | Why rejected for v1 |
|
| 20 |
|---|---|
|
| 21 |
| Pure user-upload | Highest input variance (bad scans, password-protected PDFs, partial documents). No cross-policy comparison possible — the bot only knows what the user just uploaded. |
|
| 22 |
-
| Hybrid with shared collection | User uploads polluting shared retrieval would let one bad upload poison answers for every other user. |
|
| 23 |
|
| 24 |
## Consequences
|
| 25 |
|
|
@@ -37,4 +37,4 @@ A health-insurance bot needs PDF text to ground its answers. The PDFs can come f
|
|
| 37 |
|
| 38 |
## Revisit at scale
|
| 39 |
|
| 40 |
-
Same approach, larger corpus. Add scheduled monthly refresh job that re-fetches every URL via Playwright and re-embeds any changed PDF.
|
|
|
|
| 1 |
# ADR-003: Curated corpus over user-uploaded PDFs
|
| 2 |
|
| 3 |
+
**Status:** Locked — partially superseded by [ADR-044](ADR-044-uploaded-pdf-parity.md) (2026-05-27) on the user-upload question.
|
| 4 |
**Date:** 2026-05-13
|
| 5 |
|
| 6 |
## Context
|
|
|
|
| 12 |
|
| 13 |
## Decision
|
| 14 |
|
| 15 |
+
**Curated corpus** of 148 catalogued policies across 21 insurers (current as of 2026-05-27; older docs referencing 190/19 are pre-dedup file counts) + 18 regulatory PDFs. User-uploads were initially routed to an isolated `user_uploads_quarantine` Chroma collection (v1 design). ADR-044 (2026-05-27) revised this — uploads now dual-write into both the per-session quarantine AND the global `policies` collection, so the upload becomes a first-class marketplace card with the same scorecard / premium / RAG endpoints. The 8-gate defence in `backend/security.py` is the mitigation against the "pollutes canonical retrieval" risk the original framing avoided.
|
| 16 |
|
| 17 |
## Alternatives considered
|
| 18 |
|
| 19 |
| Approach | Why rejected for v1 |
|
| 20 |
|---|---|
|
| 21 |
| Pure user-upload | Highest input variance (bad scans, password-protected PDFs, partial documents). No cross-policy comparison possible — the bot only knows what the user just uploaded. |
|
| 22 |
+
| Hybrid with shared collection | User uploads polluting shared retrieval would let one bad upload poison answers for every other user. → now what shipped via ADR-044 — the 8-gate defence + heuristic-floor + Gemini extraction chain replaces the "isolation" mitigation with an "every upload is gated, extracted, and parity-checked against the catalogued 148" mitigation. Cross-session retrieval scoping remains an open follow-up. |
|
| 23 |
|
| 24 |
## Consequences
|
| 25 |
|
|
|
|
| 37 |
|
| 38 |
## Revisit at scale
|
| 39 |
|
| 40 |
+
Same approach, larger corpus. Add scheduled monthly refresh job that re-fetches every URL via Playwright and re-embeds any changed PDF. The user-upload model has moved on — see [ADR-044](ADR-044-uploaded-pdf-parity.md) (2026-05-27) for the dual-write + 8-gate + heuristic-floor + Gemini extraction chain that supersedes the original quarantine-only design.
|
|
@@ -106,6 +106,47 @@ The choice prompt **never fires before the card lands**. This is verified by the
|
|
| 106 |
- **Gemini intermittency** — live observation shows Gemini occasionally returning unparseable output for the same prompt that succeeded earlier. NIM fallback fires automatically on Gemini failure. Cause not fully traced; tracked for follow-up.
|
| 107 |
- **Cross-session retrieval** — every upload is written to both the per-session quarantine AND the global `policies` Chroma collection (so it can become a marketplace card visible to other users). The global chunks have no `session_id` metadata, so `retrieve_policies` queries can pull chunks from other users' uploads. For the "analyse THIS upload" intent this can produce cross-policy confusion. Mitigated today by the per-session quarantine boost in retrieval; full per-session scoping is a tracked follow-up.
|
| 108 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
## Verification
|
| 110 |
|
| 111 |
- Live audit on commit `e8ccfa0` (https://rohitsar567-insurancebot.hf.space):
|
|
|
|
| 106 |
- **Gemini intermittency** — live observation shows Gemini occasionally returning unparseable output for the same prompt that succeeded earlier. NIM fallback fires automatically on Gemini failure. Cause not fully traced; tracked for follow-up.
|
| 107 |
- **Cross-session retrieval** — every upload is written to both the per-session quarantine AND the global `policies` Chroma collection (so it can become a marketplace card visible to other users). The global chunks have no `session_id` metadata, so `retrieve_policies` queries can pull chunks from other users' uploads. For the "analyse THIS upload" intent this can produce cross-policy confusion. Mitigated today by the per-session quarantine boost in retrieval; full per-session scoping is a tracked follow-up.
|
| 108 |
|
| 109 |
+
## 2026-05-27 hardening bundle (post-audit)
|
| 110 |
+
|
| 111 |
+
Post-launch audit on `e8ccfa0` surfaced five distinct defects across the upload path. All five landed in a same-day bundle (commits `2abfd01`, `e204c0e`, `2323b26`, `58e3c82`, `2a58c28`, `993bcd5`).
|
| 112 |
+
|
| 113 |
+
### H1. Status ↔ scorecard parity
|
| 114 |
+
|
| 115 |
+
- **Problem:** status endpoint reported `completeness_pct` / `overall_grade` from `build_scorecard(doc, profile=None)` without `insurer_reviews`; card endpoint used `_catalogue_scorecard(pid, None)` which folds in heuristic + curated + reviews + dedup. Numbers diverged on the same upload: status `17.4% / None`, card `47.8% / C`.
|
| 116 |
+
- **Two-bug compounding:** missing `insurer_reviews` AND reading `.overall_grade` instead of `.grade`. The dataclass attribute is `.grade`; only the wire `ScorecardResponse` renames it. So even with reviews wired in, the grade field came back `None`.
|
| 117 |
+
- **Fix:** status resolver now mirrors `/api/policies/{id}/scorecard` exact order — primary `_catalogue_scorecard(pid, None)`, fallback `build_scorecard(doc, insurer_reviews=ir, profile=None)`, read `.grade` (not `.overall_grade`). Commits `2abfd01` → `e204c0e` → `2323b26`.
|
| 118 |
+
|
| 119 |
+
### H2. Hash-cache short-circuit had the same bugs (commit `58e3c82`)
|
| 120 |
+
|
| 121 |
+
- The Tier-2 content-hash branch (sha256 match ⇒ reuse prior extraction) had the IDENTICAL three bugs the main path had (no reviews, `.overall_grade`, no insurer slug threading). Cache-hit uploads silently surfaced `comp=17.4 grade=None` even when the actual card was `47.8 / C`.
|
| 122 |
+
- **Fix:** same three-bug fix applied + new field `llm_used="hash-cache"` so the operator can see WHY no LLM ran on a given upload.
|
| 123 |
+
|
| 124 |
+
### H3. Provenance fields exposed (commit `2abfd01`)
|
| 125 |
+
|
| 126 |
+
- New status fields: `llm_used` (one of `gemini-2.5-flash#1|#2|#3`, `nim-fallback`, `hash-cache`, or `None` for failures) + `llm_response_chars`.
|
| 127 |
+
- Lets the operator verify Gemini is actually running without HF Space stdout access.
|
| 128 |
+
- Verified live: fresh upload shows `gemini-2.5-flash#1` returning 573 chars (unparseable), retry as `gemini-2.5-flash#2` returning 899 chars (parsed).
|
| 129 |
+
|
| 130 |
+
### H4. KI-330 — `view_context` → ACTIVE POLICY DIVE-IN block (commit `2a58c28`)
|
| 131 |
+
|
| 132 |
+
- Found in 2026-05-27 e2e audit: on 3 of 5 uploads, asking *"What are the waiting periods on this policy?"* got back *"Before I pull your recommendations, just a couple more (you can skip any):"*. `single_brain` pivoted to profile-building instead of answering about the policy.
|
| 133 |
+
- **Triple root cause:** (1) `ChatRequest.view_context` field declared at launch but NEVER consumed anywhere on backend, (2) `single_brain.handle_turn` signature didn't accept it, (3) frontend set `view_context.active_policy_id` only on a clicked marketplace card (openPolicy modal), never on a just-uploaded PDF.
|
| 134 |
+
- **Fix:** `handle_turn(..., view_context=None)`, new `_build_active_policy_block()` that prepends an ACTIVE POLICY DIVE-IN block to the system instruction when `view_context.active_policy_id` is set. Frontend gains an `activeUploadPid` state set when the card lands, plumbed into every chat turn's `view_context`.
|
| 135 |
+
- Verified: 9/10 grounded on post-fix audit, up from 0/10 pre-fix.
|
| 136 |
+
|
| 137 |
+
### H5. KI-331 — heuristic-floor card surfaces on LLM-fail (commit `993bcd5`)
|
| 138 |
+
|
| 139 |
+
- User caught live on `Test Policy.pdf` (8 MB) upload: after Gemini 3/3 retries failed and the 120s poll timed out, frontend pushed only a prose *"I couldn't pull a full analysis"* message, never the card — even though `record.json` already had `47.8% / grade C` from the heuristic pass that runs BEFORE the LLM fires.
|
| 140 |
+
- **Fix:** fail/timeout branch now pushes the same card-bearing assistant message as the success branch (`citations=[pid]` → inline `PolicyScorecardWidget` renders with whatever `record.json` has) + the soft caveat + the choice prompt. `setActiveUploadPid` still fires so dive-in mode activates for follow-up questions.
|
| 141 |
+
- Plus: `MAX_TRIES` bumped 40 → 50 (120s → 150s poll budget) since Gemini retry #3 sometimes lands at ~130s.
|
| 142 |
+
|
| 143 |
+
### Live verification matrix (post-bundle)
|
| 144 |
+
|
| 145 |
+
- **5 PDFs** (manipalcigna, hdfc-ergo, care-health, icici-lombard, star-health) × **7 layers** (upload, extraction, scorecard, premium baseline, premium older+PED, personalization profile, RAG grounded answer) = **35 cells, 33 green**.
|
| 146 |
+
- 2 honest misses: `Test Policy.pdf` 3/3 Gemini fails caught by heuristic floor (working as designed); one star-health "room rent" question correctly answered but keyword detector false-negative.
|
| 147 |
+
- Hash-cache parity verified on Manipal back-to-back uploads (~1s speedup, `llm_used='hash-cache'`, comp/grade equal across status + card).
|
| 148 |
+
- Playwright UI sequence audit: 5/5 checks pass (ack → card/fail → choice strictly ordered).
|
| 149 |
+
|
| 150 |
## Verification
|
| 151 |
|
| 152 |
- Live audit on commit `e8ccfa0` (https://rohitsar567-insurancebot.hf.space):
|
|
@@ -10,7 +10,7 @@ ADR numbers map 1:1 to the original `D-NNN` entries that lived in the legacy `70
|
|
| 10 |
|---|---|---|
|
| 11 |
| [ADR-001](ADR-001-vertical-slice-scope.md) | Vertical slice scope, not full platform | Locked |
|
| 12 |
| [ADR-002](ADR-002-health-category-vertical.md) | Health as the v1 vertical category | Locked |
|
| 13 |
-
| [ADR-003](ADR-003-curated-corpus.md) | Curated corpus over user-uploads | Locked |
|
| 14 |
| [ADR-004](ADR-004-hybrid-structured-vector.md) | Hybrid structured + vector retrieval | Locked |
|
| 15 |
| [ADR-005](ADR-005-nextjs-fastapi-frontend.md) | Next.js 14 + FastAPI (superseded original Streamlit pick) | Locked (revised) |
|
| 16 |
| [ADR-006](ADR-006-sarvam-first-stack.md) | Sarvam-first STT/TTS/LLM defaults | Partially superseded by ADR-019 |
|
|
|
|
| 10 |
|---|---|---|
|
| 11 |
| [ADR-001](ADR-001-vertical-slice-scope.md) | Vertical slice scope, not full platform | Locked |
|
| 12 |
| [ADR-002](ADR-002-health-category-vertical.md) | Health as the v1 vertical category | Locked |
|
| 13 |
+
| [ADR-003](ADR-003-curated-corpus.md) | Curated corpus over user-uploads | Locked (partially superseded by ADR-044) |
|
| 14 |
| [ADR-004](ADR-004-hybrid-structured-vector.md) | Hybrid structured + vector retrieval | Locked |
|
| 15 |
| [ADR-005](ADR-005-nextjs-fastapi-frontend.md) | Next.js 14 + FastAPI (superseded original Streamlit pick) | Locked (revised) |
|
| 16 |
| [ADR-006](ADR-006-sarvam-first-stack.md) | Sarvam-first STT/TTS/LLM defaults | Partially superseded by ADR-019 |
|
|
@@ -37,6 +37,7 @@ Every meaningful technical and product decision, with alternatives considered an
|
|
| 37 |
**Chose:** Pre-acquired
|
| 38 |
**Reasoning:** Removes biggest source of input variance (bad uploads); enables cross-policy comparison/recommendation; positions the corpus as a product moat vs. generic RAG-over-anything.
|
| 39 |
**Revisit at scale:** Same approach, larger corpus + scheduled refresh.
|
|
|
|
| 40 |
|
| 41 |
---
|
| 42 |
|
|
|
|
| 37 |
**Chose:** Pre-acquired
|
| 38 |
**Reasoning:** Removes biggest source of input variance (bad uploads); enables cross-policy comparison/recommendation; positions the corpus as a product moat vs. generic RAG-over-anything.
|
| 39 |
**Revisit at scale:** Same approach, larger corpus + scheduled refresh.
|
| 40 |
+
**Note:** Partially superseded by ADR-044 (2026-05-27) — user uploads now first-class via 8-gate defence + heuristic floor + Gemini extraction; the curated-corpus moat survives, the "uploads excluded" clause does not.
|
| 41 |
|
| 42 |
---
|
| 43 |
|
|
@@ -354,4 +354,10 @@ The 2026-05-14 scoreboard rows for "Voice UX", "Fact-find robotic tone", and the
|
|
| 354 |
|
| 355 |
---
|
| 356 |
|
| 357 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 354 |
|
| 355 |
---
|
| 356 |
|
| 357 |
+
### Sprint 2026-05-27 (KI-225..KI-333) — Single-brain consolidation + upload-pipeline rebuild
|
| 358 |
+
|
| 359 |
+
Single-brain rewrite (KI-225, ~5,200 LOC removed) collapses the old sales+qa+faithfulness chain into one Gemini-with-function-calling call per turn; faithfulness is now structural (brain quotes only what `retrieve_policies` + `get_policy_facts` returned). Upload pipeline rebuilt to ADR-044 parity: 8-gate defence in `backend/security.py` + heuristic floor (~65-70% post KI-332) + Gemini single-pass / multi-pass / NIM / floor extraction chain + status↔scorecard parity by construction (KI-333) + ACTIVE POLICY DIVE-IN block via `view_context.active_policy_id` (KI-330) + heuristic-floor card on LLM-fail path (KI-331). All shipped on commits 2a58c28, 993bcd5, d92f07a, 2ec48b7; currently live on `e7f799a`. No new defects introduced — the D-001..D-010 register from this audit remains the live defect set; none are upload-pipeline regressions.
|
| 360 |
+
|
| 361 |
+
---
|
| 362 |
+
|
| 363 |
+
*This file regenerates as new evidence lands. Last updated: 2026-05-27 (sprint KI-225 → KI-333 landed; ADR-044 ships).*
|
|
@@ -30,9 +30,12 @@ authority.
|
|
| 30 |
separate faithfulness judge / profile_extractor / tiered brain) was
|
| 31 |
removed — it does not exist in the codebase.
|
| 32 |
- **Retrieval** — Chroma vector store, BGE-small-en-v1.5 local 384-d
|
| 33 |
-
embeddings (`rag/retrieve.py`). Shared
|
| 34 |
-
~7.
|
| 35 |
-
|
|
|
|
|
|
|
|
|
|
| 36 |
- **Upload safety** — `backend/security.py`, 8 gates, before any embedding.
|
| 37 |
- **Data** — three repos: code (HF Space `origin` + GitHub `github`
|
| 38 |
mirror), the `rohitsar567/insurance-bot-data` HF dataset (corpus +
|
|
|
|
| 30 |
separate faithfulness judge / profile_extractor / tiered brain) was
|
| 31 |
removed — it does not exist in the codebase.
|
| 32 |
- **Retrieval** — Chroma vector store, BGE-small-en-v1.5 local 384-d
|
| 33 |
+
embeddings (`rag/retrieve.py`). Shared `policies` collection (148
|
| 34 |
+
catalogued plans across 21 insurers, ~7.3K chunks) + a per-session
|
| 35 |
+
`user_uploads_quarantine` collection (24h TTL, session-isolated). Per
|
| 36 |
+
ADR-044 (2026-05-27), uploaded PDFs dual-write into both collections —
|
| 37 |
+
the upload becomes a first-class marketplace card with the same
|
| 38 |
+
scorecard / premium / RAG endpoints as the catalogued 148.
|
| 39 |
- **Upload safety** — `backend/security.py`, 8 gates, before any embedding.
|
| 40 |
- **Data** — three repos: code (HF Space `origin` + GitHub `github`
|
| 41 |
mirror), the `rohitsar567/insurance-bot-data` HF dataset (corpus +
|
|
@@ -115,7 +115,7 @@ Every architectural decision in `70-docs/decisions.md` produces a specific artif
|
|
| 115 |
| --- | --- |
|
| 116 |
| D-001 Vertical slice scope | `kb/policies/` (10 categories only Health) |
|
| 117 |
| D-002 Health category | `40-data/corpus_urls.md` (76 health PDFs) |
|
| 118 |
-
| D-003 Curated corpus | `rag/corpus/`
|
| 119 |
| D-004 Hybrid structured + unstructured | `rag/policies.duckdb` + `rag/vectors/` |
|
| 120 |
| D-005 Next.js + FastAPI | `frontend/` + `backend/main.py` |
|
| 121 |
| D-006 Sarvam-first | `backend/providers/sarvam_*.py` |
|
|
|
|
| 115 |
| --- | --- |
|
| 116 |
| D-001 Vertical slice scope | `kb/policies/` (10 categories only Health) |
|
| 117 |
| D-002 Health category | `40-data/corpus_urls.md` (76 health PDFs) |
|
| 118 |
+
| D-003 Curated corpus + user-upload pipeline (ADR-044, 2026-05-27) | `rag/corpus/` for catalogued + `backend/uploaded_docs.py` + `backend/security.py` for user uploads (8-gate defence, heuristic floor + Gemini extraction with multi-pass for big PDFs) |
|
| 119 |
| D-004 Hybrid structured + unstructured | `rag/policies.duckdb` + `rag/vectors/` |
|
| 120 |
| D-005 Next.js + FastAPI | `frontend/` + `backend/main.py` |
|
| 121 |
| D-006 Sarvam-first | `backend/providers/sarvam_*.py` |
|
|
@@ -2,20 +2,28 @@
|
|
| 2 |
|
| 3 |
_Auto-generated. Source modules: `backend/security.py` + `backend/faithfulness.py`._
|
| 4 |
|
| 5 |
-
## Upload security —
|
| 6 |
|
| 7 |
Every PDF uploaded via `/api/upload-policy` runs through these gates before
|
| 8 |
-
indexing.
|
|
|
|
| 9 |
|
| 10 |
| # | Gate | Check |
|
| 11 |
| --- | --- | --- |
|
| 12 |
-
| 1 |
|
| 13 |
| 2 | Content quality | ≥1,500 chars text; ≥3 pages; ≥1 insurance keyword match (catches "garbage PDF" uploads) |
|
| 14 |
-
| 3 | Prompt injection |
|
| 15 |
-
| 4 |
|
| 16 |
-
| 5 | IP rate limit | 10 uploads/hour/IP (per X-Forwarded-For or peer IP) |
|
| 17 |
-
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## Hallucination defense — 5 gates (runtime, per-turn)
|
| 21 |
|
|
|
|
| 2 |
|
| 3 |
_Auto-generated. Source modules: `backend/security.py` + `backend/faithfulness.py`._
|
| 4 |
|
| 5 |
+
## Upload security — 8 gates
|
| 6 |
|
| 7 |
Every PDF uploaded via `/api/upload-policy` runs through these gates before
|
| 8 |
+
indexing. Pipeline lives in `backend/uploaded_docs.py` + `backend/security.py`,
|
| 9 |
+
governed by ADR-044 (2026-05-27). Failure logs to `logs/upload_blocks.jsonl`.
|
| 10 |
|
| 11 |
| # | Gate | Check |
|
| 12 |
| --- | --- | --- |
|
| 13 |
+
| 1 | File mechanics | Magic bytes `%PDF`; size 5KB-25MB; `%%EOF` present; dangerous PDF features (`/JavaScript`, `/Launch`, `/OpenAction`, `/EmbeddedFile`, `/SubmitForm`, `/AA`, `/RichMedia`, `/Movie`, `/Sound`, `/GoToR`); embedded executable signatures (Windows PE, Linux ELF, Mach-O, Java class, shell, HTML/JS, PHP) |
|
| 14 |
| 2 | Content quality | ≥1,500 chars text; ≥3 pages; ≥1 insurance keyword match (catches "garbage PDF" uploads) |
|
| 15 |
+
| 3 | Prompt injection | Regex sweep for "ignore previous instructions", "system prompt reveal", jailbreak markers, role-takeover patterns, im_start/im_end tokens |
|
| 16 |
+
| 4 | Per-session rate limit | 5 uploads/hour/session; 200 chunks/session lifetime |
|
| 17 |
+
| 5 | Per-IP rate limit | 10 uploads/hour/IP (per X-Forwarded-For or peer IP) |
|
| 18 |
+
| 6 | Encrypted / locked PDF reject | Refuse any PDF that is password-protected or has restrictive permissions blocking text extraction |
|
| 19 |
+
| 7 | Page-count ceiling | Reject PDFs with >200 pages |
|
| 20 |
+
| 8 | Hash dedupe + reject-cache | Re-uploads of an already-accepted PDF are deduped; re-uploads of a previously-rejected hash are short-circuited |
|
| 21 |
+
|
| 22 |
+
Beyond the 8, a **UIN net-new check** + **PDF-text fuzzy match** against the
|
| 23 |
+
catalogued 148 also run — uploads that match an existing catalogued policy
|
| 24 |
+
short-circuit to the catalogued card.
|
| 25 |
+
|
| 26 |
+
All gates run for EVERY upload. Block on any failure; the audit trail captures the reason set. See README §2.8 and `70-docs/60-decisions/ADR-044-uploaded-pdf-parity.md` for the dual-write model and the heuristic-floor / Gemini extraction chain.
|
| 27 |
|
| 28 |
## Hallucination defense — 5 gates (runtime, per-turn)
|
| 29 |
|