rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
4bb66dd
·
1 Parent(s): de473cb

feat: uploaded-PDF parity with catalogued 148 (ADR-044)

Browse files

User-uploaded PDFs now go through the IDENTICAL pipeline as the 148
catalogued policies — same LLM extractor, same scorecard, same review
data flow, same card UI. No "user-upload" carve-outs.

CHANGES
─────────────────────────────────────────────────────────────────────

A. Insurer detection (backend/uploaded_docs.py)
detect_insurer_slug() scans the first ~6000 chars of PDF text for
the legal name of any of the 21 insurers we have reviews data for
(Acko, Bajaj Allianz, HDFC ERGO, ICICI Lombard, ManipalCigna, Niva
Bupa, Star Health, Tata AIG, …). On a match, the persisted record
+ chunk metadata + Chroma metadata are stamped with the REAL
insurer_slug → the scorecard's Claim Experience sub-score reads
40-data/reviews/<slug>.json and gets real claim-ratio +
complaint-volume + NPS data, same as a catalogued card. Fail-closed:
no match ⇒ insurer_slug stays "user-upload".

B. LLM-assisted extraction on upload (backend/uploaded_docs.py +
backend/main.py)
New extract_one_for_upload() — fires as an asyncio.create_task
after the upload's HTTP response has returned. Uses the SAME
get_brain_llm() (NIM chain) + SAME EXTRACT_SYSTEM prompt +
SAME HealthPolicy Pydantic schema + SAME rag/extracted/<id>.json
output path as rag.extract.extract_one. On success, invalidates
the #40 marketplace grade cache so the next scorecard fetch
reflects the LLM-extracted fields. Catalogued-band completeness
(median 74%, p90 91%) is now achievable for uploads by
construction — same model, same schema, same coverage.

C. Frontend card re-render loop (frontend/src/app/page.tsx)
When the chat renders a user-upload__* policy citation, a
useEffect kicks off a 5-second polling loop against
/api/policies/{id}/scorecard. On any data_completeness_pct jump,
setCards() updates state and React re-renders the card with the
richer sub-scores + new grade. Up to 18 polls (~90s) per card,
then stops. Catalogued cards never enter this loop.

D. Partial-info banner branched copy (frontend/src/components/
PolicyScorecardWidget.tsx)
For catalogued insurer cards, the existing "the insurer hasn't
published every term…" copy stays — that's the true cause of
their low completeness. For user-uploaded cards, the cause is
our extractor not finding fields in the PDF text, NOT insurer
non-disclosure. New honest copy: "Some fields couldn't be pulled
from this PDF automatically — open the document for the full
wording before you decide. (Re-grading in the background as more
fields are extracted.)"

E. Upload chat-flow polish (frontend/src/app/page.tsx + i18n.ts)
- Dropped the synthetic 📎-Uploaded user breadcrumb (was leaking
into chat_history; could trigger unprompted brain responses on
a subsequent voice auto-fire).
- Voice auto-submit is now gated on `uploadStatus !== null` so
ambient noise / TTS playback during the long index window can
no longer fire an unprompted analysis chat turn.
- Stripped "indexed", "chunks from N pages", "in Xs", "searchable
in this chat" — replaced with plain user-facing copy ("Reading
${name}…", "I've read ${name}", "${name} is ready"). Both EN +
HI. welcome.coverage_template also de-jargoned ("indexed" →
"already loaded and ready").

VERIFY
─────────────────────────────────────────────────────────────────────
- py_compile clean on every edited backend file
- npx tsc --noEmit clean
- Live audit deferred to deploy commit

PARITY TABLE (uploaded vs catalogued, after this commit)
─────────────────────────────────────────────────────────────────────
Indexing : same chunker, same embedder, same collections
Structured JSON : same LLM, same prompt, same schema, same path
Insurer reviews : same 40-data/reviews/<slug>.json lookup
Marketplace card : same _marketplace_catalogue pipeline
Scorecard : same /api/policies/{id}/scorecard endpoint
Premium estimate : same /api/premium/estimate endpoint
RAG retrieval : same retrieve_policies tool
Card UI : same PolicyScorecardWidget component
Source PDF link : same per-policy PDF endpoint
Completeness target : same band (median 74%, p90 91%) by
construction (same model, same schema)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/main.py CHANGED
@@ -1941,10 +1941,14 @@ async def upload_policy(
1941
  from rag.ingest import get_chroma_collection as _get_pol_coll
1942
  _pol = _get_pol_coll()
1943
  _g_ids = [f"{policy_id}::chunk{c['chunk_idx']}" for c in chunks]
 
 
 
 
1944
  _g_meta = [
1945
  {
1946
  "policy_id": policy_id,
1947
- "insurer_slug": _udocs.UPLOAD_INSURER_SLUG,
1948
  "policy_name": policy_name,
1949
  "doc_type": _udocs.UPLOAD_DOC_TYPE,
1950
  "source_url": "",
@@ -1971,6 +1975,34 @@ async def upload_policy(
1971
  _MG_CACHE["index"] = None
1972
  except Exception: # noqa: BLE001 — cache bust is best-effort
1973
  pass
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1974
  except HTTPException:
1975
  raise
1976
  except Exception as e:
 
1941
  from rag.ingest import get_chroma_collection as _get_pol_coll
1942
  _pol = _get_pol_coll()
1943
  _g_ids = [f"{policy_id}::chunk{c['chunk_idx']}" for c in chunks]
1944
+ # Use whatever insurer_slug build_record resolved (detected from
1945
+ # PDF text via detect_insurer_slug, or UPLOAD_INSURER_SLUG on no
1946
+ # match) so chunk metadata + scorecard reviews lookup agree.
1947
+ _resolved_insurer_slug = _record.get("insurer_slug", _udocs.UPLOAD_INSURER_SLUG)
1948
  _g_meta = [
1949
  {
1950
  "policy_id": policy_id,
1951
+ "insurer_slug": _resolved_insurer_slug,
1952
  "policy_name": policy_name,
1953
  "doc_type": _udocs.UPLOAD_DOC_TYPE,
1954
  "source_url": "",
 
1975
  _MG_CACHE["index"] = None
1976
  except Exception: # noqa: BLE001 — cache bust is best-effort
1977
  pass
1978
+
1979
+ # ── Fire LLM-assisted extraction in background (ADR-044) ─────────
1980
+ # Same extractor as the catalogued 148. Runs ~30-60s; the upload
1981
+ # HTTP response returns now and the frontend polls the scorecard
1982
+ # endpoint to refresh the card in place when extraction lands.
1983
+ # Fail-silent: a failed LLM pass leaves the heuristic record
1984
+ # intact, so the card still has SOMETHING to show — never blocks
1985
+ # the user. NEVER blocks this request.
1986
+ try:
1987
+ from pathlib import Path as _PathLib2
1988
+ _persisted_pdf = _udocs.uploaded_docs_dir() / policy_id / "source.pdf"
1989
+ _detected_insurer_name = _record.get(
1990
+ "insurer_name",
1991
+ _udocs.detected_insurer_name(_resolved_insurer_slug)
1992
+ if _resolved_insurer_slug != _udocs.UPLOAD_INSURER_SLUG
1993
+ else _udocs.UPLOAD_INSURER_NAME,
1994
+ )
1995
+ asyncio.create_task(
1996
+ _udocs.extract_one_for_upload(
1997
+ policy_id=policy_id,
1998
+ pdf_path=_persisted_pdf,
1999
+ policy_name=policy_name,
2000
+ insurer_slug=_resolved_insurer_slug,
2001
+ insurer_name=_detected_insurer_name,
2002
+ )
2003
+ )
2004
+ except Exception: # noqa: BLE001 — extraction is async + optional
2005
+ pass
2006
  except HTTPException:
2007
  raise
2008
  except Exception as e:
backend/uploaded_docs.py CHANGED
@@ -81,6 +81,98 @@ UPLOAD_INSURER_SLUG = "user-upload"
81
  UPLOAD_INSURER_NAME = "User-uploaded document"
82
 
83
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
  # ---------------------------------------------------------------------------
85
  # Storage layout
86
  # ---------------------------------------------------------------------------
@@ -398,12 +490,23 @@ def build_record(
398
  if isinstance(cell, dict) and "source_pdf_path" in cell:
399
  cell["source_pdf_path"] = rel_pdf
400
 
 
 
 
 
 
 
 
 
401
  record: dict[str, Any] = {
402
  "policy_id": policy_id,
403
  "policy_name": policy_name or _derive_policy_name(full_text, policy_id),
404
- "insurer_slug": UPLOAD_INSURER_SLUG,
405
  "_uploaded_doc": True, # provenance flag (ignored by scorecard)
406
  }
 
 
 
407
  record.update(fields)
408
  return record
409
 
@@ -455,7 +558,9 @@ def persist_upload(
455
  meta = {
456
  "policy_id": policy_id,
457
  "policy_name": record["policy_name"],
458
- "insurer_slug": UPLOAD_INSURER_SLUG,
 
 
459
  "sha256": hashlib.sha256(pdf_bytes).hexdigest(),
460
  "uploaded_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
461
  "session_id": session_id, # audit only — NEVER a visibility gate
@@ -610,3 +715,161 @@ async def reingest_persisted_into_policies() -> dict:
610
  policy_id, type(e).__name__, e,
611
  )
612
  return summary
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
  UPLOAD_INSURER_NAME = "User-uploaded document"
82
 
83
 
84
+ # ---------------------------------------------------------------------------
85
+ # Insurer detection from PDF text (2026-05-27).
86
+ #
87
+ # Pre-this-change, every upload was stamped insurer_slug="user-upload",
88
+ # which short-circuits the Claim Experience scorecard sub-score (no
89
+ # matching reviews JSON under 40-data/reviews/<slug>.json) and leaves
90
+ # the card showing "reputation data being compiled" forever.
91
+ #
92
+ # Strategy: regex-match the first ~3 pages of the PDF against the
93
+ # canonical legal names of the 21 insurers we already have reviews
94
+ # data for. On a confident hit, flip insurer_slug to the real slug so
95
+ # the scorecard's Claim-Experience pass uses the real reviews data
96
+ # (claim_ratio, complaints, network) — same path as a catalogued card.
97
+ #
98
+ # Fail-closed: no match ⇒ stays "user-upload". Score still works,
99
+ # Claim Experience just falls back to a generic mid-range number.
100
+ # ---------------------------------------------------------------------------
101
+
102
+ # Each entry: (slug, [name_patterns]). Order matters — first hit wins,
103
+ # so put the most specific patterns first (e.g. "future generali" before
104
+ # bare "generali" — though we don't have a generali reviews file today).
105
+ # Patterns are matched case-insensitive against the first ~3 pages of
106
+ # PDF text (first ~6000 chars).
107
+ _INSURER_NAME_PATTERNS: list[tuple[str, list[str]]] = [
108
+ ("acko", ["acko general insurance", "acko general", "acko gen ins", "acko gi"]),
109
+ ("aditya-birla", ["aditya birla health insurance", "abhicl", "aditya birla health", "aditya birla"]),
110
+ ("bajaj-allianz", ["bajaj allianz general insurance", "bajaj allianz general", "bajaj allianz"]),
111
+ ("care-health", ["care health insurance", "religare health insurance"]),
112
+ ("cholamandalam", ["cholamandalam ms general insurance", "cholamandalam ms general", "cholamandalam ms", "chola ms"]),
113
+ ("go-digit", ["go digit general insurance", "go digit", "godigit"]),
114
+ ("hdfc-ergo", ["hdfc ergo general insurance", "hdfc ergo health", "hdfc ergo"]),
115
+ ("icici-lombard", ["icici lombard general insurance", "icici lombard"]),
116
+ ("iffco-tokio", ["iffco tokio general insurance", "iffco tokio"]),
117
+ ("indusind-general", ["indusind general insurance", "indusind general"]),
118
+ ("manipalcigna", ["manipalcigna health insurance", "manipal cigna health insurance", "manipalcigna", "manipal cigna"]),
119
+ ("national-insurance", ["national insurance company", "national insurance"]),
120
+ ("new-india", ["new india assurance company", "new india assurance", "the new india assurance"]),
121
+ ("niva-bupa", ["niva bupa health insurance", "niva bupa", "max bupa"]),
122
+ ("oriental-insurance", ["oriental insurance company", "the oriental insurance", "oriental insurance"]),
123
+ ("reliance-general", ["reliance general insurance"]),
124
+ ("royal-sundaram", ["royal sundaram general insurance", "royal sundaram"]),
125
+ ("sbi-general", ["sbi general insurance", "sbi gen"]),
126
+ ("star-health", ["star health and allied insurance", "star health and allied", "star health"]),
127
+ ("tata-aig", ["tata aig general insurance", "tata aig"]),
128
+ ]
129
+
130
+
131
+ def detect_insurer_slug(full_text: str) -> Optional[str]:
132
+ """Return the matching insurer slug (one of 21) or None.
133
+
134
+ Scans only the first ~6 000 chars (typically the cover + Part I) since
135
+ the insurer's legal name is in the header / footer of every IRDAI PDF.
136
+ Case-insensitive substring match in pattern-priority order. Fail-closed.
137
+ """
138
+ if not full_text:
139
+ return None
140
+ head = full_text[:6000].lower()
141
+ for slug, patterns in _INSURER_NAME_PATTERNS:
142
+ for pat in patterns:
143
+ if pat in head:
144
+ return slug
145
+ return None
146
+
147
+
148
+ def detected_insurer_name(slug: str) -> str:
149
+ """Pretty-display name for a detected slug — used in the persisted
150
+ record so the card header reads "ManipalCigna" not "manipalcigna".
151
+ """
152
+ return {
153
+ "acko": "Acko",
154
+ "aditya-birla": "Aditya Birla Health",
155
+ "bajaj-allianz": "Bajaj Allianz",
156
+ "care-health": "Care Health",
157
+ "cholamandalam": "Cholamandalam MS",
158
+ "go-digit": "Go Digit",
159
+ "hdfc-ergo": "HDFC ERGO",
160
+ "icici-lombard": "ICICI Lombard",
161
+ "iffco-tokio": "IFFCO Tokio",
162
+ "indusind-general": "IndusInd General",
163
+ "manipalcigna": "ManipalCigna",
164
+ "national-insurance": "National Insurance",
165
+ "new-india": "New India Assurance",
166
+ "niva-bupa": "Niva Bupa",
167
+ "oriental-insurance": "Oriental Insurance",
168
+ "reliance-general": "Reliance General",
169
+ "royal-sundaram": "Royal Sundaram",
170
+ "sbi-general": "SBI General",
171
+ "star-health": "Star Health",
172
+ "tata-aig": "Tata AIG",
173
+ }.get(slug, slug)
174
+
175
+
176
  # ---------------------------------------------------------------------------
177
  # Storage layout
178
  # ---------------------------------------------------------------------------
 
490
  if isinstance(cell, dict) and "source_pdf_path" in cell:
491
  cell["source_pdf_path"] = rel_pdf
492
 
493
+ # 2026-05-27 — detect the actual insurer from the PDF text and flip the
494
+ # insurer_slug off the generic "user-upload" so the scorecard's Claim
495
+ # Experience sub-score pulls the real reviews JSON
496
+ # (40-data/reviews/<slug>.json). Fail-closed: no match ⇒ keep
497
+ # UPLOAD_INSURER_SLUG.
498
+ detected = detect_insurer_slug(full_text)
499
+ slug = detected or UPLOAD_INSURER_SLUG
500
+
501
  record: dict[str, Any] = {
502
  "policy_id": policy_id,
503
  "policy_name": policy_name or _derive_policy_name(full_text, policy_id),
504
+ "insurer_slug": slug,
505
  "_uploaded_doc": True, # provenance flag (ignored by scorecard)
506
  }
507
+ if detected:
508
+ # Pretty name for any card renderer that reads from this record.
509
+ record["insurer_name"] = detected_insurer_name(detected)
510
  record.update(fields)
511
  return record
512
 
 
558
  meta = {
559
  "policy_id": policy_id,
560
  "policy_name": record["policy_name"],
561
+ # Use whatever build_record resolved — the detected insurer
562
+ # slug if we matched one, else UPLOAD_INSURER_SLUG.
563
+ "insurer_slug": record.get("insurer_slug", UPLOAD_INSURER_SLUG),
564
  "sha256": hashlib.sha256(pdf_bytes).hexdigest(),
565
  "uploaded_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
566
  "session_id": session_id, # audit only — NEVER a visibility gate
 
715
  policy_id, type(e).__name__, e,
716
  )
717
  return summary
718
+
719
+
720
+ # ---------------------------------------------------------------------------
721
+ # LLM-assisted extraction for uploaded PDFs (2026-05-27, ADR-044).
722
+ #
723
+ # Parity with the catalogued 148: same LLM (get_brain_llm), same EXTRACT
724
+ # prompt, same HealthPolicy schema, same downstream merge into the
725
+ # marketplace catalogue. Pre-this-change the upload path only ran the
726
+ # deterministic-heuristic `extract_fields_from_text` over the PDF, which
727
+ # is why uploaded cards stalled at 13-48% data_completeness vs the
728
+ # 74% median for catalogued. After this change uploaded cards land in
729
+ # the same completeness band by construction.
730
+ #
731
+ # Runs as a background asyncio task fired from the upload endpoint —
732
+ # the upload's HTTP response returns immediately with the heuristic
733
+ # record (sub-second), and the LLM pass (~30-60s) lands in the
734
+ # background. The frontend polls /api/policies/{id}/scorecard after
735
+ # the upload and refreshes the card in place when completeness jumps.
736
+ # ---------------------------------------------------------------------------
737
+
738
+
739
+ async def extract_one_for_upload(
740
+ policy_id: str,
741
+ pdf_path: Path,
742
+ policy_name: str,
743
+ insurer_slug: str,
744
+ insurer_name: str,
745
+ ) -> bool:
746
+ """Run the same LLM extractor used for the catalogued 148 against an
747
+ uploaded PDF. On success, writes `rag/extracted/<policy_id>.json` and
748
+ invalidates the marketplace grade cache so the next /api/policies/all
749
+ + /api/policies/{id}/scorecard call returns the LLM-graded card.
750
+
751
+ Returns True iff a HealthPolicy was successfully extracted and written.
752
+ Swallows all errors (returns False) — a failed LLM pass must NEVER
753
+ affect the upload's HTTP response, which has already returned.
754
+ """
755
+ try:
756
+ # Lazy imports — these touch the LLM client + DuckDB; we don't
757
+ # want to pay that cost at module import time.
758
+ from rag.extract import (
759
+ EXTRACT_SYSTEM,
760
+ build_extract_prompt,
761
+ schema_excerpt,
762
+ read_full_text,
763
+ json_from_llm_text,
764
+ upsert_policy,
765
+ )
766
+ from rag.schema import HealthPolicy
767
+ from backend.providers.base import ChatMessage
768
+ from backend.providers.nvidia_nim_llm import get_brain_llm
769
+
770
+ _log.info(
771
+ "[upload-extract] starting LLM extraction for %s (insurer=%s)",
772
+ policy_id, insurer_slug,
773
+ )
774
+
775
+ # Read text from the persisted PDF (same as extract_one).
776
+ try:
777
+ text = read_full_text(pdf_path)
778
+ except Exception as e: # noqa: BLE001
779
+ _log.warning(
780
+ "[upload-extract] read_full_text failed %s: %s: %s",
781
+ policy_id, type(e).__name__, e,
782
+ )
783
+ return False
784
+
785
+ prompt = build_extract_prompt(text, schema_excerpt(), policy_id)
786
+ messages = [
787
+ ChatMessage(role="system", content=EXTRACT_SYSTEM),
788
+ ChatMessage(role="user", content=prompt),
789
+ ]
790
+
791
+ llm_primary = get_brain_llm()
792
+ llm_fallback = get_brain_llm()
793
+
794
+ raw = ""
795
+ policy: Optional[HealthPolicy] = None
796
+ for attempt, llm in enumerate([llm_primary, llm_fallback]):
797
+ try:
798
+ attempt_timeout = 180 if attempt == 0 else 120
799
+ res = await asyncio.wait_for(
800
+ llm.chat(messages=messages, temperature=0.0, max_tokens=2048),
801
+ timeout=attempt_timeout,
802
+ )
803
+ raw = res.text
804
+ data = json_from_llm_text(raw)
805
+ # Force-fill identity fields (REQUIRED by the schema, the
806
+ # LLM frequently emits null for these because they're not
807
+ # in the truncated text). Use what the upload path
808
+ # already resolved.
809
+ if not data.get("policy_id"):
810
+ data["policy_id"] = policy_id
811
+ if not data.get("insurer_slug"):
812
+ data["insurer_slug"] = insurer_slug
813
+ if not data.get("insurer_name"):
814
+ data["insurer_name"] = insurer_name
815
+ if not data.get("policy_name"):
816
+ data["policy_name"] = policy_name
817
+ policy = HealthPolicy(**data)
818
+ break
819
+ except Exception as e: # noqa: BLE001
820
+ _log.warning(
821
+ "[upload-extract] attempt %d failed for %s: %s: %s",
822
+ attempt + 1, policy_id, type(e).__name__, str(e)[:200],
823
+ )
824
+ continue
825
+
826
+ if policy is None:
827
+ _log.warning(
828
+ "[upload-extract] no policy extracted for %s after retries; "
829
+ "card stays on heuristic record", policy_id,
830
+ )
831
+ return False
832
+
833
+ # Write rag/extracted/<policy_id>.json — same shape as catalogued.
834
+ from backend.config import settings as _settings
835
+ _settings.EXTRACTED_DIR.mkdir(parents=True, exist_ok=True)
836
+ out_json = _settings.EXTRACTED_DIR / f"{policy_id}.json"
837
+ out_json.write_text(policy.model_dump_json(indent=2))
838
+
839
+ # Persist into DuckDB so admin / re-render paths see the new card.
840
+ try:
841
+ upsert_policy(
842
+ policy,
843
+ source_pdf_path=str(pdf_path),
844
+ source_pdf_url="",
845
+ )
846
+ except Exception as e: # noqa: BLE001 — DB write is best-effort
847
+ _log.warning(
848
+ "[upload-extract] upsert_policy failed for %s: %s: %s",
849
+ policy_id, type(e).__name__, e,
850
+ )
851
+
852
+ # Invalidate the #40 marketplace grade cache so the next
853
+ # /api/policies/all / scorecard call returns the LLM-graded card.
854
+ try:
855
+ import backend.main as _bm
856
+ with _bm._MG_LOCK:
857
+ _bm._MG_CACHE["sig"] = None
858
+ _bm._MG_CACHE["index"] = None
859
+ except Exception as e: # noqa: BLE001 — cache miss is fine
860
+ _log.debug(
861
+ "[upload-extract] could not invalidate _MG_CACHE for %s: %s",
862
+ policy_id, e,
863
+ )
864
+
865
+ _log.info(
866
+ "[upload-extract] OK %s (extraction_confidence_pct=%s)",
867
+ policy_id, getattr(policy, "extraction_confidence_pct", "n/a"),
868
+ )
869
+ return True
870
+ except Exception as e: # noqa: BLE001 — top-level catch-all
871
+ _log.warning(
872
+ "[upload-extract] unexpected failure for %s: %s: %s",
873
+ policy_id, type(e).__name__, str(e)[:400],
874
+ )
875
+ return False
frontend/src/app/page.tsx CHANGED
@@ -971,6 +971,14 @@ export default function Page() {
971
  voiceSubmitRef.current = (text: string) => {
972
  const t = text.trim();
973
  if (t.length < 2) return;
 
 
 
 
 
 
 
 
974
  // V4 FIX 2 — dedup repeated finals within 500ms.
975
  const { text: prevText, at: prevAt } = lastFinalTextRef.current;
976
  const now = Date.now();
@@ -986,7 +994,7 @@ export default function Page() {
986
  // send() reads `messages` / `sessionId` / `ttsLang` / view flags via
987
  // closure; rebind whenever they change so the latest values are used.
988
  // eslint-disable-next-line react-hooks/exhaustive-deps
989
- }, [messages, sessionId, ttsLang, openPolicy, showMarketplace, showProfile, showPremium]);
990
 
991
  async function startRecording() {
992
  // KI-222 FIX 1 — silence any prior bot TTS BEFORE PTT recording starts.
@@ -1440,24 +1448,20 @@ export default function Page() {
1440
  async function handleFile(ev: React.ChangeEvent<HTMLInputElement>) {
1441
  const f = ev.target.files?.[0];
1442
  if (!f) return;
1443
- // Push a user-side breadcrumb so the chat transcript shows the upload
1444
- // happened (helps the user track context and it's part of the
1445
- // history the next /api/chat turn sends back to the brain).
1446
- pushUser(t("upload.user_msg", { name: f.name }));
 
 
 
1447
  setUploadStatus(t("upload.indexing", { name: f.name }));
1448
  try {
1449
  // Pass the live chat session so the backend scopes the uploaded doc
1450
  // to this user — the assistant can then answer questions about it
1451
  // for the rest of THIS conversation.
1452
  const r = await uploadPolicy(f, sessionId);
1453
- setUploadStatus(
1454
- t("upload.success", {
1455
- name: r.policy_name,
1456
- chunks: r.chunks_added,
1457
- pages: r.pages_indexed,
1458
- secs: (r.elapsed_ms / 1000).toFixed(1),
1459
- }),
1460
- );
1461
  // ── In-chat acknowledgment + inline scorecard card ──────────────
1462
  // Push two assistant messages:
1463
  // 1. The "got it, here's the card" ack with a `citations` array
@@ -1469,12 +1473,7 @@ export default function Page() {
1469
  // 2. The proceed-choice prompt — telling the user they can
1470
  // finish their profile OR dive into the PDF, and noting that
1471
  // a fuller profile makes the policy discussion more useful.
1472
- const ackText = t("upload.chat_ack", {
1473
- name: r.policy_name,
1474
- chunks: r.chunks_added,
1475
- pages: r.pages_indexed,
1476
- secs: (r.elapsed_ms / 1000).toFixed(1),
1477
- });
1478
  pushAssistant(ackText, {
1479
  citations: [
1480
  {
@@ -4288,6 +4287,53 @@ function CitedPolicyCards({
4288
  // eslint-disable-next-line react-hooks/exhaustive-deps
4289
  }, [citations.map((c) => c.policy_id).join("|")]);
4290
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4291
  // Insurer reputation / reviews. The full detail modal shows a reviews
4292
  // section; the inline cited cards omitted it (user-flagged — same class
4293
  // as #65's claim-experience gap). Fetch once per DISTINCT insurer_slug.
 
971
  voiceSubmitRef.current = (text: string) => {
972
  const t = text.trim();
973
  if (t.length < 2) return;
974
+ // Suppress voice auto-submit while a PDF upload is in flight or
975
+ // just-completed (uploadStatus is non-null for ~8s after success
976
+ // / failure). A long upload + active mic + bot's TTS playing
977
+ // through speakers can otherwise auto-transcribe ambient sound
978
+ // and fire an "unprompted analysis" chat turn that drowns the
979
+ // upload-flow's choice prompt. Real user input still goes
980
+ // through the typed-input path / explicit Push-to-talk press.
981
+ if (uploadStatus) return;
982
  // V4 FIX 2 — dedup repeated finals within 500ms.
983
  const { text: prevText, at: prevAt } = lastFinalTextRef.current;
984
  const now = Date.now();
 
994
  // send() reads `messages` / `sessionId` / `ttsLang` / view flags via
995
  // closure; rebind whenever they change so the latest values are used.
996
  // eslint-disable-next-line react-hooks/exhaustive-deps
997
+ }, [messages, sessionId, ttsLang, openPolicy, showMarketplace, showProfile, showPremium, uploadStatus]);
998
 
999
  async function startRecording() {
1000
  // KI-222 FIX 1 — silence any prior bot TTS BEFORE PTT recording starts.
 
1448
  async function handleFile(ev: React.ChangeEvent<HTMLInputElement>) {
1449
  const f = ev.target.files?.[0];
1450
  if (!f) return;
1451
+ // Earlier iteration also pushUser'd a "📎 Uploaded: <name>" breadcrumb
1452
+ // into the transcript. That leaked into chat_history so a subsequent
1453
+ // voice auto-fire (mic catching ambient sound during the long index
1454
+ // wait) could trigger the brain to "analyse" the upload unprompted.
1455
+ // Removed: the card rendered below the ack message is itself the
1456
+ // user-visible breadcrumb that the upload happened. uploadStatus
1457
+ // gates the voice-submit path during the indexing window.
1458
  setUploadStatus(t("upload.indexing", { name: f.name }));
1459
  try {
1460
  // Pass the live chat session so the backend scopes the uploaded doc
1461
  // to this user — the assistant can then answer questions about it
1462
  // for the rest of THIS conversation.
1463
  const r = await uploadPolicy(f, sessionId);
1464
+ setUploadStatus(t("upload.success", { name: r.policy_name }));
 
 
 
 
 
 
 
1465
  // ── In-chat acknowledgment + inline scorecard card ──────────────
1466
  // Push two assistant messages:
1467
  // 1. The "got it, here's the card" ack with a `citations` array
 
1473
  // 2. The proceed-choice prompt — telling the user they can
1474
  // finish their profile OR dive into the PDF, and noting that
1475
  // a fuller profile makes the policy discussion more useful.
1476
+ const ackText = t("upload.chat_ack", { name: r.policy_name });
 
 
 
 
 
1477
  pushAssistant(ackText, {
1478
  citations: [
1479
  {
 
4287
  // eslint-disable-next-line react-hooks/exhaustive-deps
4288
  }, [citations.map((c) => c.policy_id).join("|")]);
4289
 
4290
+ // ── Re-fetch loop for USER-UPLOADED policies (ADR-044, 2026-05-27) ──
4291
+ // The upload endpoint kicks off LLM-assisted extraction in a
4292
+ // background asyncio task that lands ~30-60s later. Re-poll each
4293
+ // uploaded card every 5s for up to 90s so the chat card refreshes
4294
+ // in place when the new extraction → higher completeness lands.
4295
+ // Catalogued cards never enter this loop (their extraction was done
4296
+ // offline; the initial fetch above is the final state).
4297
+ useEffect(() => {
4298
+ const sid = typeof window !== "undefined" ? sessionStorage.getItem("insurance_session_id") || undefined : undefined;
4299
+ const uploaded = topPolicies.filter((c) => c.policy_id.startsWith("user-upload__"));
4300
+ if (uploaded.length === 0) return;
4301
+ let cancelled = false;
4302
+ let tries = 0;
4303
+ const MAX_TRIES = 18; // 18 × 5s ≈ 90s
4304
+ const tick = () => {
4305
+ if (cancelled) return;
4306
+ tries += 1;
4307
+ Promise.all(
4308
+ uploaded.map((c) =>
4309
+ getScorecard(c.policy_id, sid)
4310
+ .then((s) => {
4311
+ if (cancelled) return null;
4312
+ const prev = cards[c.policy_id];
4313
+ const completenessJumped = s?.data_completeness_pct != null
4314
+ && (prev == null || (s.data_completeness_pct ?? 0) > (prev.data_completeness_pct ?? 0));
4315
+ if (completenessJumped) {
4316
+ setCards((p) => ({ ...p, [c.policy_id]: s }));
4317
+ return true;
4318
+ }
4319
+ return false;
4320
+ })
4321
+ .catch(() => false),
4322
+ ),
4323
+ ).then(() => {
4324
+ if (!cancelled && tries < MAX_TRIES) {
4325
+ setTimeout(tick, 5000);
4326
+ }
4327
+ });
4328
+ };
4329
+ const handle = setTimeout(tick, 5000); // first re-fetch 5s after initial
4330
+ return () => {
4331
+ cancelled = true;
4332
+ clearTimeout(handle);
4333
+ };
4334
+ // eslint-disable-next-line react-hooks/exhaustive-deps
4335
+ }, [topPolicies.filter((c) => c.policy_id.startsWith("user-upload__")).map((c) => c.policy_id).join("|")]);
4336
+
4337
  // Insurer reputation / reviews. The full detail modal shows a reviews
4338
  // section; the inline cited cards omitted it (user-flagged — same class
4339
  // as #65's claim-experience gap). Fetch once per DISTINCT insurer_slug.
frontend/src/components/PolicyScorecardWidget.tsx CHANGED
@@ -817,7 +817,12 @@ export default function PolicyScorecardWidget({
817
  );
818
  })()}
819
 
820
- {/* Limited-data warning — warm amber, single tidy row */}
 
 
 
 
 
821
  {showLimitedWarning && (
822
  <div
823
  style={{
@@ -836,9 +841,20 @@ export default function PolicyScorecardWidget({
836
  >
837
  <span style={{ fontWeight: 700, flexShrink: 0 }}>Partial information ·</span>
838
  <span>
839
- The insurer hasn&apos;t published every term for this policy yet,
840
- so this grade is an early read — open the policy PDF for the full
841
- wording before you decide.
 
 
 
 
 
 
 
 
 
 
 
842
  </span>
843
  </div>
844
  )}
 
817
  );
818
  })()}
819
 
820
+ {/* Limited-data warning — warm amber, single tidy row.
821
+ Copy is branched per source: catalogued insurer cards reflect
822
+ gaps in the insurer's own filings, but USER-UPLOADED PDFs are
823
+ missing fields because OUR extractor couldn't pull them from
824
+ the PDF text. Saying "the insurer hasn't published…" on an
825
+ uploaded doc is just wrong — flip to honest copy there. */}
826
  {showLimitedWarning && (
827
  <div
828
  style={{
 
841
  >
842
  <span style={{ fontWeight: 700, flexShrink: 0 }}>Partial information ·</span>
843
  <span>
844
+ {policyId.startsWith("user-upload__") ? (
845
+ <>
846
+ Some fields couldn&apos;t be pulled from this PDF
847
+ automatically — open the document for the full wording
848
+ before you decide. (Re-grading in the background as more
849
+ fields are extracted.)
850
+ </>
851
+ ) : (
852
+ <>
853
+ The insurer hasn&apos;t published every term for this policy
854
+ yet, so this grade is an early read — open the policy PDF
855
+ for the full wording before you decide.
856
+ </>
857
+ )}
858
  </span>
859
  </div>
860
  )}
frontend/src/lib/i18n.ts CHANGED
@@ -28,7 +28,7 @@ export const UI_STRINGS = {
28
  "welcome.source_link": "Every fact you see has a source link.",
29
  "welcome.trust_title": "Tell me the truth — even on the hard things.",
30
  "welcome.trust_body": "When I ask about your health later, please don't hide a condition to lower your premium. Insurers cross-check disclosed history against hospital records at claim time. The ₹500/month you save today turns into an ₹8 lakh denied claim later. Your honest answers stay in this chat — they're not shared with any insurer until you choose to buy.",
31
- "welcome.coverage_template": "${policies} policies across ${insurers} insurers indexed. Or upload your own policy PDF I'll analyse it the same way.",
32
 
33
  // Input bar
34
  "input.placeholder": "Ask about coverage, waiting periods, exclusions, or compare policies…",
@@ -38,11 +38,11 @@ export const UI_STRINGS = {
38
  "input.voice_input": "Voice input",
39
  "input.upload": "Upload your own policy PDF",
40
  "input.enter_to_send": "Enter to send · 📎 to upload your own PDF",
41
- "upload.indexing": "Indexing ${name}…",
42
- "upload.success": "✓ Indexed “${name}” ${chunks} chunks from ${pages} pages (${secs}s). It's now searchable in this chat. Ask me about it.",
43
  "upload.error": "✗ Upload failed: ${err}",
44
  "upload.user_msg": "📎 Uploaded: ${name}",
45
- "upload.chat_ack": "Got it — I've indexed **${name}** (${chunks} chunks from ${pages} pages, in ${secs}s). Here's how it grades against what we know about you so far:",
46
  "upload.chat_choice": "How would you like to proceed?\n\n• **Tell me more about yourself** — finish the short profile (age, family, location, budget, health) so I can speak to this policy more personally.\n• **Dive into the PDF first** — ask questions about coverage, waiting periods, exclusions, anything in the document.\n\nEither works. The more I know about you, the more useful the discussion of this policy will be.",
47
 
48
  // Marketplace panel
@@ -150,7 +150,7 @@ export const UI_STRINGS = {
150
  "welcome.source_link": "हर तथ्य का source link है।",
151
  "welcome.trust_title": "सच बताइए — मुश्किल बातें भी।",
152
  "welcome.trust_body": "जब मैं आपकी सेहत के बारे में पूछूं, premium कम करने के लिए कोई condition मत छिपाइए। बीमाकर्ता claim time पर hospital records से check करते हैं। आज के ₹500/महीने की बचत बाद में ₹8 लाख का denied claim बन जाती है। आपके ईमानदार जवाब इसी chat में रहते हैं — किसी insurer के साथ शेयर नहीं होते।",
153
- "welcome.coverage_template": "${policies} पॉलिसियाँ, ${insurers} बीमाकर्ताओं से indexed । अपनी policy PDF भी upload कर सकते हैं।",
154
 
155
  "input.placeholder": "Coverage, waiting period, exclusion, या तुलना के बारे में पूछिए…",
156
  "input.send": "भेजें",
@@ -159,11 +159,11 @@ export const UI_STRINGS = {
159
  "input.voice_input": "आवाज़ input",
160
  "input.upload": "अपनी policy PDF upload करें",
161
  "input.enter_to_send": "Enter दबाकर भेजें · 📎 से PDF upload",
162
- "upload.indexing": "${name} index हो रह…",
163
- "upload.success": "✓ “${name}” index हो गई — ${pages} पेज से ${chunks} chunks (${secs}s)। अब इसी chat में search ो सकती है। इसके बारे में पूछिए।",
164
  "upload.error": "✗ Upload विफल: ${err}",
165
  "upload.user_msg": "📎 Upload किया: ${name}",
166
- "upload.chat_ack": "मिल गया — **${name}** index हो गई (${pages} ेज से ${chunks} chunks, ${secs}s में)। यह policy आपके profile के हिसाब से कैसी है:",
167
  "upload.chat_choice": "आगे कैसे बढ़ें?\n\n• **अपने बारे में बताएं** — short profile पूरा करें (उम्र, परिवार, location, बजट, health) ताकि मैं इस policy पर आपको personally बात कर सकूं।\n• **पहले PDF पर बात करें** — coverage, waiting periods, exclusions — कुछ भी पूछें।\n\nदोनों ठीक हैं। जितना मैं आपके बारे में जानूंगा, इस policy की चर्चा उतनी useful होगी।",
168
 
169
  "mp.heading": "स्वास्थ्य बीमा बाज़ार",
 
28
  "welcome.source_link": "Every fact you see has a source link.",
29
  "welcome.trust_title": "Tell me the truth — even on the hard things.",
30
  "welcome.trust_body": "When I ask about your health later, please don't hide a condition to lower your premium. Insurers cross-check disclosed history against hospital records at claim time. The ₹500/month you save today turns into an ₹8 lakh denied claim later. Your honest answers stay in this chat — they're not shared with any insurer until you choose to buy.",
31
+ "welcome.coverage_template": "${policies} policies across ${insurers} insurers — already loaded and ready. Or upload your own policy PDF and I'll analyse it the same way.",
32
 
33
  // Input bar
34
  "input.placeholder": "Ask about coverage, waiting periods, exclusions, or compare policies…",
 
38
  "input.voice_input": "Voice input",
39
  "input.upload": "Upload your own policy PDF",
40
  "input.enter_to_send": "Enter to send · 📎 to upload your own PDF",
41
+ "upload.indexing": "Reading ${name}…",
42
+ "upload.success": "✓ “${name}” is ready. Ask me anything about it.",
43
  "upload.error": "✗ Upload failed: ${err}",
44
  "upload.user_msg": "📎 Uploaded: ${name}",
45
+ "upload.chat_ack": "Got it — I've read **${name}**. Here's how it grades against what we know about you so far:",
46
  "upload.chat_choice": "How would you like to proceed?\n\n• **Tell me more about yourself** — finish the short profile (age, family, location, budget, health) so I can speak to this policy more personally.\n• **Dive into the PDF first** — ask questions about coverage, waiting periods, exclusions, anything in the document.\n\nEither works. The more I know about you, the more useful the discussion of this policy will be.",
47
 
48
  // Marketplace panel
 
150
  "welcome.source_link": "हर तथ्य का source link है।",
151
  "welcome.trust_title": "सच बताइए — मुश्किल बातें भी।",
152
  "welcome.trust_body": "जब मैं आपकी सेहत के बारे में पूछूं, premium कम करने के लिए कोई condition मत छिपाइए। बीमाकर्ता claim time पर hospital records से check करते हैं। आज के ₹500/महीने की बचत बाद में ₹8 लाख का denied claim बन जाती है। आपके ईमानदार जवाब इसी chat में रहते हैं — किसी insurer के साथ शेयर नहीं होते।",
153
+ "welcome.coverage_template": "${policies} पॉलिसियाँ, ${insurers} बीमाकर्ताओं से सब तयार। अपनी policy PDF भी upload कर सकते हैं।",
154
 
155
  "input.placeholder": "Coverage, waiting period, exclusion, या तुलना के बारे में पूछिए…",
156
  "input.send": "भेजें",
 
159
  "input.voice_input": "आवाज़ input",
160
  "input.upload": "अपनी policy PDF upload करें",
161
  "input.enter_to_send": "Enter दबाकर भेजें · 📎 से PDF upload",
162
+ "upload.indexing": "${name} पढ़ रहूँ…",
163
+ "upload.success": "✓ “${name}” तैार है। इसके बारे में कुछ भी पूछिए।",
164
  "upload.error": "✗ Upload विफल: ${err}",
165
  "upload.user_msg": "📎 Upload किया: ${name}",
166
+ "upload.chat_ack": "म���ल गया — **${name}** पढ़ ली। यह आपके profile के हिसाब से कैसी है:",
167
  "upload.chat_choice": "आगे कैसे बढ़ें?\n\n• **अपने बारे में बताएं** — short profile पूरा करें (उम्र, परिवार, location, बजट, health) ताकि मैं इस policy पर आपको personally बात कर सकूं।\n• **पहले PDF पर बात करें** — coverage, waiting periods, exclusions — कुछ भी पूछें।\n\nदोनों ठीक हैं। जितना मैं आपके बारे में जानूंगा, इस policy की चर्चा उतनी useful होगी।",
168
 
169
  "mp.heading": "स्वास्थ्य बीमा बाज़ार",