feat(upload-extract): Tier-1 Gemini stability + new ADR-044 + README rewrites

UPLOADED_DOCS Gemini retry hardening (extract_one_for_upload):
- Gemini retry count 1 → 3 with jittered exp backoffs (2s/4s/8s ±25%),
mirroring ADR-042's _TRANSIENT_RETRY_BACKOFFS_STICKY pattern that
handles Gemini's 429/5xx tail in single_brain.
- NIM moved to FINAL fallback (after Gemini exhausts retries).
- Raw LLM responses persisted to
UPLOADED_DOCS_DIR/<pid>/llm_raw_<n>_<label>.txt on every attempt
(success or failure) — gives the operator visibility into WHY a
Gemini call failed (was it 429, truncation mid-emission, markdown
fence around the JSON, schema mismatch?). Previously a black box.

DOCS:
- New 70-docs/60-decisions/ADR-044-uploaded-pdf-parity.md — the
architectural record for the entire upload-PDF parity effort
(LLM extraction + insurer detection + status endpoint + sequence
lock + backfill). Includes verification fingerprints from the
live e8ccfa0 audit.
- README §2.1 user-journey diagram extended with the new upload
sub-flow (UP → UPIDX → UPCARD → UPCHOICE) and supporting prose.
- README §4.4 "Uploaded-PDF field extraction is deterministic-
heuristic, not LLM" REWRITTEN — now describes the two-pass
heuristic-floor + LLM-augment pipeline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

70-docs/60-decisions/ADR-044-uploaded-pdf-parity.md

@@ -0,0 +1,118 @@
+# ADR-044 — Uploaded-PDF parity with catalogued 148 + locked upload sequence
+
+**Date:** 2026-05-27
+**Status:** Accepted
+**Related:** Extends ADR-022 (conversational profile updates), ADR-040 (Gemini primary), ADR-043 (no cross-session recall). Does not supersede any.
+**Commits:** `4bb66dd` (initial parity), `245310d` (asyncio import hotfix), `88e54a8` (backfill), `24349be` (PTT voice gate), `9ec9eae` (Header import hotfix), `7ef3ca3` → `fd30ceb` → `52b0b5d` → `835f745` → `e8ccfa0` (LLM-extraction iteration), `dfaa4d6` → `3dcbe9b` (sequence lock).
+
+## Context
+
+Pre-this-ADR the upload pipeline produced a card that the user (rightly) said *"lacks all data as required in parity with the existing policy pdf cards"*:
+
+- Card displayed only ~13–48% of decision-critical fields (vs the 74% median completeness on the catalogued 148).
+- `insurer_slug` was hardcoded to `"user-upload"`, so the Claim Experience sub-score got no real IRDAI claim-ratio data — the inline card showed *"Reviews: reputation data being compiled"* indefinitely (placeholder text with no backend that ever populates it).
+- The chat sequence around an upload was broken: ack message + choice prompt fired in parallel BEFORE the card was ready, and voice auto-fire during the long indexing window dropped unprompted "please upload" or analysis messages into chat.
+- Old uploads (uploaded before the new pipeline was wired) never got upgraded.
+
+User directive (2026-05-27): *"Whatever is done for the 148 policies pre-populated, exact same level of usability needs to be there for user-uploaded PDFs. No less."* And the chat sequence: *"the card should not be presented until the card summary and the detailed LLM extraction and full field population is done. And until the card is presented, no further questions should be asked."*
+
+## Decision
+
+Three coordinated changes, all in service of catalogued-grade parity for live uploads.
+
+### D1. LLM-assisted extraction pipeline for live uploads
+
+Identical extractor as the catalogued 148:
+
+| Step | Same as catalogued 148 |
+|---|---|
+| Chunking | ✅ `rag/ingest.py`, ~500-token overlap, BGE-small-en-v1.5 384-d embeddings |
+| Embedding into Chroma | ✅ Per-session `user_uploads_quarantine` + global `policies` collection |
+| LLM extraction | ✅ Same `get_brain_llm()` (Gemini 2.5-flash primary, NIM chain fallback), same `EXTRACT_SYSTEM` prompt, same `HealthPolicy` Pydantic schema |
+| Structured JSON output | ✅ Same `rag/extracted/<policy_id>.json` path |
+| Marketplace inclusion | ✅ Same `_marketplace_catalogue` Pass-1 + Pass-2 pipeline; uploads become first-class cards (`UPLOADED_DOCS_DIR/<pid>/`) |
+| Scorecard | ✅ Same `/api/policies/{id}/scorecard` endpoint, no `user-upload__` branches |
+| Premium estimate | ✅ Same `/api/premium/estimate` endpoint |
+| RAG chat answers | ✅ Same `retrieve_policies` tool, same faithfulness guard |
+| Card UI | ✅ Same `PolicyScorecardWidget` component, no source-aware branches |
+| Source-PDF link | ✅ Persisted PDF at `UPLOADED_DOCS_DIR/<pid>/source.pdf`, served via the same per-policy PDF endpoint |
+
+The only difference from the catalogued 148 is that catalogued PDFs went through **multiple extraction passes + human review + curated overlay** (`40-data/policy_facts/<insurer>__<product>.json`). Live uploads run **one extraction pass with no human review**, but with two safety nets:
+
+- **Heuristic baseline floor** — `build_record()` in `backend/uploaded_docs.py` synchronously regex-extracts ~30–50% of fields during the upload HTTP call. This is the floor; on LLM extraction failure the card still has real data. Same shape as the curated layer (`{value, source_pdf_path, source_quote, _confidence}`).
+- **LLM-into-heuristic merge** — when `extract_one_for_upload` completes, the LLM output is merged INTO `record.json` (LLM wins per-field where non-empty, heuristic stays where LLM was silent). This is the same "extracted + curated overlay" model the catalogued 148 use.
+
+### D2. Insurer detection from PDF text
+
+`backend/uploaded_docs.py::detect_insurer_slug()` scans the first ~6 000 chars of the PDF text against the 21 insurer name patterns we have reviews data for (acko, aditya-birla, bajaj-allianz, care-health, cholamandalam, go-digit, hdfc-ergo, icici-lombard, iffco-tokio, indusind-general, manipalcigna, national-insurance, new-india, niva-bupa, oriental-insurance, reliance-general, royal-sundaram, sbi-general, star-health, tata-aig). On a hit, `insurer_slug` flips off the generic `"user-upload"` to the real slug. The Claim Experience sub-score then reads the corresponding `40-data/reviews/<slug>.json` — same path as a catalogued card. Fail-closed: no match ⇒ stays `"user-upload"` (the generic insurer reputation isn't fabricated, the sub-score just falls back to a neutral score).
+
+Live verification: Sarvah Param.pdf (a ManipalCigna product) → detected as `manipalcigna` → Claim Experience score = 83/100 with verbatim signals `["cashless supported", "99.0% CSR (IRDAI 2023-24)"]`.
+
+### D3. New upload status endpoint + frontend polling
+
+- **New `_UPLOAD_EXTRACTION_STATUS` in-memory dict** in `backend/uploaded_docs.py` tracks per-upload state: `pending → running → complete | failed`.
+- **`extract_one_for_upload` writes status** at every phase + final `completeness_pct` and `overall_grade`.
+- **New `GET /api/upload/extraction-status/{policy_id}`** exposes the live dict. Returns `status="unknown"` for unrecognised policy_ids so the client can stop polling.
+- **Frontend `handleFile` polls** every 3 s for up to 120 s. Card-bearing assistant message + choice prompt fire only after the poll sees `complete` or `failed`/timeout.
+
+### D4. Locked chat sequence
+
+Per user directive (verbatim): *"until the card summary and the detailed LLM extraction and full field population is done, the card should not be presented. And until the card is presented, no further questions should be asked."* Implemented:
+
+```
+1. User picks file (PDF input)
+2. setUploadStatus(t("upload.indexing"))         ← amber banner above composer
+   setExtractionInFlight(true)                    ← gates EVERY send-path
+3. POST /api/upload-policy                        ← ~10-60s
+4. pushAssistant(ack_reading)                     ← "Got it — reading X, ~30-60s"
+   [NO card yet, NO choice prompt yet]
+5. Poll /api/upload/extraction-status every 3s, up to 120s
+   [During this entire wait window, EVERYTHING is gated:
+    - Send button:    disabled={busy || !input.trim() || extractionInFlight}
+    - Textarea:       disabled={busy || extractionInFlight}
+    - PDF button:     disabled={busy || extractionInFlight}
+    - send() function: early-return on extractionInFlight (last-line guard)
+    - voiceSubmitRef:  guarded
+    - PTT/Sarvam path: guarded
+   ]
+6. On status="complete":
+     pushAssistant(card_ready_message, { citations: [{policy_id, ...}] })
+     [card renders inline below this bubble]
+     pushAssistant(choice_prompt)
+   On status="failed" / timeout:
+     pushAssistant(extraction_failed_message)
+     pushAssistant(choice_prompt)
+7. setExtractionInFlight(false)                   ← controls re-enable
+8. User picks "finish profile" / "dive into PDF" → normal chat continues
+```
+
+The choice prompt **never fires before the card lands**. This is verified by the in-code ordering — both branches of step 6 push the choice prompt AFTER the prior message.
+
+### D5. Backfill for legacy uploads
+
+`backend/uploaded_docs.py::backfill_extractions(force=False)` iterates `UPLOADED_DOCS_DIR`, skips any policy_id that already has `rag/extracted/<pid>.json` (unless `force=True`), and runs `extract_one_for_upload` for the rest. Fired as a fire-and-forget `asyncio.create_task` from the `@app.on_event("startup")` hook so every container boot upgrades any old upload that was persisted before this pipeline was wired. Also exposed as `POST /api/admin/upload/reextract?force=<bool>` for on-demand operator triggering.
+
+## Consequences
+
+- **Live upload cards reach catalogued-grade depth** by construction: same LLM, same schema, same scorecard endpoint. Verified live: Sarvah Param.pdf upload → grade C, score 65/100, 6 sub-scores all populated with verbatim signals, real IRDAI claim data via manipalcigna insurer detection.
+- **Heuristic baseline is a hard floor** — even if Gemini 5xxs / parses fail, the card still produces a real grade C off the 47.8% heuristic completeness.
+- **No race conditions in the chat flow** — Send button + every voice path are gated by `extractionInFlight`, and the `send()` function has a last-line guard so even a programmatic future caller can't bypass.
+- **Backward-compatible for old uploads** — backfill upgrades anything persisted under `UPLOADED_DOCS_DIR/` before this pipeline was wired.
+- **Operator visibility** — `/api/upload/extraction-status/{pid}` shows live state to the frontend AND can be polled by the operator for debugging.
+
+## Caveats / open follow-ups
+
+- **Single LLM pass per upload** — catalogued 148 had multiple extraction passes + human review. A non-standard PDF that the heuristic also can't read well will produce a card at the heuristic floor (40–50% completeness) without a grade lift from the LLM. Multi-pass extraction is a known follow-up (task #109).
+- **Gemini intermittency** — live observation shows Gemini occasionally returning unparseable output for the same prompt that succeeded earlier. NIM fallback fires automatically on Gemini failure. Cause not fully traced; tracked for follow-up.
+- **Cross-session retrieval** — every upload is written to both the per-session quarantine AND the global `policies` Chroma collection (so it can become a marketplace card visible to other users). The global chunks have no `session_id` metadata, so `retrieve_policies` queries can pull chunks from other users' uploads. For the "analyse THIS upload" intent this can produce cross-policy confusion. Mitigated today by the per-session quarantine boost in retrieval; full per-session scoping is a tracked follow-up.
+
+## Verification
+
+- Live audit on commit `e8ccfa0` (https://rohitsar567-insurancebot.hf.space):
+  - Fresh upload of Sarvah Param.pdf via `/api/upload-policy`
+  - Extraction status: pending → running → complete in ~12 s
+  - Scorecard endpoint: `grade=C, score=65, completeness=52.2%, insurer_slug=manipalcigna`
+  - Six sub-scores all populated with verbatim signals
+  - Claim Experience score 83 with `["cashless supported", "99.0% CSR (IRDAI 2023-24)"]` — real IRDAI claim data, sourced from `40-data/reviews/manipalcigna.json` via the detected insurer slug
+- Heuristic baseline floor verified: when Gemini fails on a fresh upload, the card still serves `grade=C, score=64, completeness=47.8%` via the heuristic record
+- Sequence lock verified live: choice prompt always appears AFTER card_ready_message OR extraction_failed_message, never before

README.md

@@ -133,11 +133,15 @@ flowchart TD
     EXPLORE -->|"Compare"| CMP["⚖️ Compare up to 4 plans side by side · full scorecard per plan"]
     EXPLORE -->|"Browse"| MKT["📚 Browse the full indexed marketplace"]
     EXPLORE -->|"Ask"| QA["💬 Ask follow-up questions — answered only from the actual documents"]
-    EXPLORE -->|"My own policy"| UP["📄 Upload your own policy PDF — ask about YOUR document (private to your session)"]
+    EXPLORE -->|"My own policy"| UP["📄 Upload your own policy PDF"]
+    UP --> UPIDX["⏳ Quick ack — 'Reading it through, ~30–60 s'<br/>(everything in chat is gated while the analysis runs)"]
+    UPIDX --> UPCARD["📊 Inline scorecard card with FULL data:<br/>grade letter · 6 sub-scores · verbatim signals · insurer reputation"]
+    UPCARD --> UPCHOICE{"How would you like to proceed?"}
+    UPCHOICE -->|"Finish profile"| TELL
+    UPCHOICE -->|"Dive into the PDF"| QA
     CMP --> PREM
     MKT --> PREM
     QA --> PREM
-    UP --> PREM
     PREM["💸 A live premium estimate that updates as you change your profile"] --> DONE["✅ Decide with confidence — no lead capture, no commission bias"]
     VOICE["🎙️ Optional the whole way: speak instead of type — it speaks the answers back"] -.-> TELL
     VOICE -.-> QA
@@ -160,6 +164,15 @@ memory; closing the tab forgets you (privacy-by-design, see ADR-043).
 - **Branches from the shortlist.** Compare side by side, browse the full
   marketplace, ask follow-up questions, or upload your *own* policy PDF
   and ask about your document (kept private to your session).
+- **Upload-PDF flow is a staged sequence** (ADR-044, 2026-05-27):
+  upload → bot says *"reading it through, ~30–60 s"* → all chat input is
+  gated during the wait (Send button, textarea, voice paths all blocked
+  so nothing can interrupt the staging) → bot pushes the inline
+  scorecard card with FULL extracted data once the LLM pass lands → bot
+  then asks whether you'd like to finish your profile or dive into the
+  PDF. The card is the same shape as any catalogued policy card — six
+  sub-scores, verbatim signals, real claim-settlement data when the
+  insurer is recognised.
 - **Live premium.** Updates as you change the profile.
 - **Decision.** No lead capture and no commission bias — the path ends at
   *decide*, not at a sales handoff.
@@ -915,10 +928,29 @@ These are real and stated up front rather than buried:
   curated/extracted baseline. Treat uploads as session-scoped. An
   operator/abuse prune endpoint exists (`POST /api/admin/uploaded-docs/
   prune`, password-gated) to remove a persisted upload by id or prefix.
-- **Uploaded-PDF field extraction is deterministic-heuristic, not LLM.**
-  A standard IRDAI-format wording yields a real (non-sentinel) grade; a
-  scanned-image or non-standard PDF with little extractable text honestly
-  shows the data-starved sentinel rather than a fabricated grade.
+- **Uploaded-PDF field extraction is LLM-assisted, with a deterministic
+  heuristic floor (ADR-044, 2026-05-27).** Every upload runs through two
+  passes:
+  - **Heuristic baseline** — regex + keyword extraction over the PDF text,
+    runs synchronously inside the upload HTTP call (sub-second), populates
+    common fields like waiting periods and room-rent rule. Yields
+    ~30–50 % data_completeness.
+  - **LLM-assisted extraction** — fires as a background asyncio task
+    after the upload returns. Same `get_brain_llm()` chain the catalogued
+    148 use offline (Gemini 2.5-flash primary, NVIDIA NIM fallback);
+    same `EXTRACT_SYSTEM` prompt; same `HealthPolicy` Pydantic schema;
+    output written to `rag/extracted/<policy_id>.json` and merged INTO
+    the persisted `record.json` (LLM values override where present,
+    heuristic stays where the LLM was silent). ~10–60 s.
+  The frontend polls `GET /api/upload/extraction-status/<policy_id>`
+  during the wait and renders the inline scorecard card ONLY after the
+  LLM pass either completes or hits its 120 s timeout. The card is
+  catalogued-grade — same `PolicyScorecardWidget`, same six sub-scores,
+  same insurer reputation data (because `detect_insurer_slug` matches
+  the PDF's legal name against the 21 known insurer slugs we have
+  reviews data for and flips `insurer_slug` off the generic `user-upload`
+  on a hit). On any LLM-pass failure the heuristic floor still produces a
+  real grade, never a fabricated one or a data-starved sentinel.
 - **Live (BETA) voice mode** uses the browser's in-built speech
   recognition and is labelled unstable; **push-to-talk** is the reliable
   path (warm-armed mic + pre-roll so the first word is never clipped, and

backend/uploaded_docs.py

@@ -853,38 +853,51 @@ async def extract_one_for_upload(
             ChatMessage(role="user", content=prompt),
         ]
 
-        # Gemini-primary (user directive 2026-05-27) — same brain the
-        # live chat uses (ADR-040), native JSON-mode via
-        # response_mime_type=application/json. NIM stays as the
-        # transport-level fallback for: GOOGLE_API_KEY missing, 5xx
-        # outage, or quota exhaustion. Key fix vs the earlier broken
-        # swap: max_tokens bumped 2048 → 8192 so the full HealthPolicy
-        # JSON (23 nested fields + verbatim quotes) doesn't get
-        # truncated mid-emission, which was the actual failure mode.
-        llm_primary = GoogleGeminiLLM(timeout=180.0)
-        llm_fallback = get_brain_llm()
-
+        # Tier-1 Gemini-stability hardening (ADR-044, 2026-05-27):
+        #   1. Bumped retry count from 1 → 3 on the Gemini primary path
+        #      with jittered exp backoffs (2s/4s/8s ±25%). Mirrors the
+        #      _TRANSIENT_RETRY_BACKOFFS_STICKY pattern from single_brain
+        #      (ADR-042) that proved effective for Gemini's 429/5xx tail.
+        #   2. NIM is the FINAL fallback after Gemini exhausts retries.
+        #   3. Raw LLM responses are captured to disk on each failed
+        #      attempt — UPLOADED_DOCS_DIR/<pid>/llm_raw_<n>.txt — so the
+        #      operator can SEE why a Gemini call failed (was it a 429,
+        #      truncated mid-emission, returned with markdown fences,
+        #      etc.). Previously the failure was a black box.
+        import random as _random
+        _GEMINI_BACKOFFS = (2.0, 4.0, 8.0)
+        _GEMINI_JITTER = 0.25
+        def _jit(b: float) -> float:
+            return b * _random.uniform(1 - _GEMINI_JITTER, 1 + _GEMINI_JITTER)
+
+        llm_gemini = GoogleGeminiLLM(timeout=180.0)
+        llm_nim = get_brain_llm()
         raw = ""
         policy: Optional[HealthPolicy] = None
-        for attempt, (llm, label) in enumerate(
-            [(llm_primary, "gemini-2.5-flash"), (llm_fallback, "nim-fallback")]
-        ):
+        attempts: list[tuple[object, str]] = [
+            (llm_gemini, "gemini-2.5-flash#1"),
+            (llm_gemini, "gemini-2.5-flash#2"),
+            (llm_gemini, "gemini-2.5-flash#3"),
+            (llm_nim, "nim-fallback"),
+        ]
+        for attempt, (llm, label) in enumerate(attempts):
             try:
-                attempt_timeout = 180 if attempt == 0 else 120
+                # Backoff BEFORE every attempt after the first Gemini try.
+                # First attempt: no wait. Subsequent Gemini attempts: jittered
+                # exp. NIM fallback: no extra backoff (Gemini already gave up).
+                if 1 <= attempt <= len(_GEMINI_BACKOFFS):
+                    _bo = _jit(_GEMINI_BACKOFFS[attempt - 1])
+                    _log.info(
+                        "[upload-extract] sleeping %.1fs before %s retry",
+                        _bo, label,
+                    )
+                    await asyncio.sleep(_bo)
+                attempt_timeout = 180 if label.startswith("gemini") else 120
                 chat_kwargs = {
                     "messages": messages,
                     "temperature": 0.0,
                     "max_tokens": 8192,
                 }
-                # NOTE — we DON'T set response_format here. The
-                # EXTRACT_SYSTEM prompt already mandates JSON-only output
-                # ("Single object. No whitespace beyond what's needed.").
-                # Setting Gemini's responseMimeType=application/json was
-                # silently producing payloads that json_from_llm_text /
-                # HealthPolicy schema validation rejected. Letting the
-                # model emit JSON natively (and using json_from_llm_text's
-                # tolerant front-strip of fences + <think> blocks) is the
-                # path the catalogued 148 used successfully.
                 res = await asyncio.wait_for(
                     llm.chat(**chat_kwargs),
                     timeout=attempt_timeout,
@@ -894,6 +907,12 @@ async def extract_one_for_upload(
                     "[upload-extract] %s returned %d chars; parsing JSON…",
                     label, len(raw or ""),
                 )
+                # Persist raw response for ops visibility (always — both
+                # successful + failed parses get a copy).
+                try:
+                    (_doc_dir(policy_id) / f"llm_raw_{attempt + 1}_{label.replace('#','-')}.txt").write_text(raw or "")
+                except Exception:
+                    pass
                 data = json_from_llm_text(raw)
                 # Force-fill identity fields (REQUIRED by the schema, the
                 # LLM frequently emits null for these because they're not