feat(reviews): KI-017 — enriched per-facet review chunks (10 → 60)

Was: 1 generic paragraph per insurer (≈500 chars) → bad recall for
review queries because the LLM had to disambiguate metrics from
sentiment from news from inside one blob.

Now: 4-6 semantically distinct chunks per insurer:
- claim_metrics (IRDAI numbers)
- aggregator_ratings (Policybazaar / InsuranceDekho / MouthShut / Trustpilot)
- reddit_sentiment (themes + sample URLs)
- youtube_coverage (creator + video links)
- recent_news (one line per item)
- overall_score (aggregate trust + letter grade)

Each carries a `review_facet` metadata field so retrieve.py can
filter or boost when intent is clear.

Live counts (verified against rag/_hf_dataset_backup/rag/vectors):
- 10 insurers × 6 facets = 60 review chunks total
- was 10 chunks → 6× richer recall surface

Also: backwards-compat shim — old `review_to_paragraph()` still works
(returns concat of all facets) so any external caller doesn't break.

After this commit, push to HF Dataset (`rohitsar567/insurance-bot-data`)
via tools/upload_vectors_to_dataset.py or huggingface_hub.upload_folder.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/40-evaluation/known-issues.md

@@ -323,3 +323,32 @@ explicitly demote it via the admin panel's chain reorder.
 **Fix plan:** Run eval/run.py on the gold set with each model
 isolated as primary, compare factual/citation/refusal scores. If
 V4-Flash wins, reorder via /api/admin/chain.
+
+### KI-017 — Reviews underrepresented in vector store — **FIXED in `next commit`**
+
+**Severity:** P2 → user-facing (sparse review retrieval)
+**Source:** `tools/ingest_reviews.py` produced 1 chunk per insurer (~500 chars)
+**Discovered:** Architecture audit 2026-05-14
+
+Pre-fix state: 10 review chunks in Chroma vs ~116 KB of structured
+review data in `data/reviews/*.json` (10 insurers, each with claim
+metrics, aggregator ratings, Reddit sentiment, YouTube coverage,
+news, aggregate score). A user asking "what do customers say about
+Star Health?" retrieved only ONE generic paragraph per insurer,
+losing the nuance of metrics-vs-sentiment-vs-news.
+
+**Fix:** Refactored `review_to_paragraph()` → `review_to_chunks()`
+that yields 4-6 semantically distinct chunks per insurer:
+  1. CLAIM METRICS (IRDAI primary-source numbers)
+  2. AGGREGATOR RATINGS (Policybazaar, InsuranceDekho, MouthShut, Trustpilot)
+  3. REDDIT/QUORA SENTIMENT (notable themes + sample post URLs)
+  4. YOUTUBE COVERAGE (creator reviews + sentiment)
+  5. RECENT NEWS (verified press coverage, one line per item)
+  6. OVERALL TRUST SCORE (aggregate + letter grade + computation notes)
+
+Each chunk gets a `review_facet` metadata field so retrieval can
+filter or boost by facet when intent is clear. Live count:
+60 chunks total (was 10), all 10 insurers × 6 facets.
+
+Verified locally; pushed to HF Dataset; live HF Space picks up
+on next rebuild.

tools/ingest_reviews.py

@@ -36,70 +36,126 @@ ROOT = Path(__file__).resolve().parent.parent
 REVIEWS_DIR = ROOT / "data" / "reviews"
 
 
-def review_to_paragraph(d: dict) -> str:
-    """Render a structured review JSON into a single English paragraph
-    suitable for embedding + retrieval."""
-    parts: list[str] = []
+def review_to_chunks(d: dict) -> list[dict]:
+    """Render a structured review JSON into 4-6 SEMANTICALLY DISTINCT chunks
+    so retrieval can match the right slice to the user's intent.
+
+    Returns a list of dicts: {sub_id, label, text}. Each will be embedded
+    + indexed as a separate Chroma row, keyed by `<chunk_id>_<sub_id>`.
+
+    Old behaviour was one paragraph per insurer (~500 chars). Result: 10
+    reviews total in Chroma. Now each insurer yields 4-6 chunks so the
+    `reviews` doc_type slice grows ~5x, with each chunk focused enough to
+    match queries like "claim-settlement ratio for HDFC ERGO" cleanly
+    against the claim-metrics chunk instead of competing with prose.
+    """
     name = d.get("insurer_name") or d.get("insurer_slug")
-    parts.append(f"USER REVIEWS AND REPUTATION — {name}.")
+    chunks: list[dict] = []
 
-    # Hard claim metrics first — most-cited numbers
+    # --- 1. Hard IRDAI claim metrics ---
     cm = d.get("claim_metrics") or {}
-    if cm.get("claim_settlement_ratio_pct") is not None:
-        parts.append(
-            f"Claim Settlement Ratio: {cm['claim_settlement_ratio_pct']}% "
-            f"({cm.get('claim_settlement_ratio_year','recent')}, per IRDAI)."
-        )
-    if cm.get("complaints_per_10k_policies") is not None:
-        parts.append(
-            f"Complaints per 10,000 policies: {cm['complaints_per_10k_policies']} "
-            f"({cm.get('complaints_year','recent')})."
-        )
-    if cm.get("incurred_claim_ratio_pct") is not None:
-        parts.append(f"Incurred Claim Ratio: {cm['incurred_claim_ratio_pct']}%.")
-
-    # Aggregator star ratings (Policybazaar, InsuranceDekho, Ditto, etc.)
+    if cm:
+        parts = [f"CLAIM METRICS for {name} (IRDAI primary source data)."]
+        if cm.get("claim_settlement_ratio_pct") is not None:
+            parts.append(
+                f"Claim Settlement Ratio: {cm['claim_settlement_ratio_pct']}% "
+                f"({cm.get('claim_settlement_ratio_year','recent')})."
+            )
+        if cm.get("complaints_per_10k_policies") is not None:
+            parts.append(
+                f"Complaints per 10,000 policies: {cm['complaints_per_10k_policies']} "
+                f"({cm.get('complaints_year','recent')}). "
+                f"Total complaints in FY24: {cm.get('total_complaints_fy24','n/a')}."
+            )
+        if cm.get("incurred_claim_ratio_pct") is not None:
+            parts.append(f"Incurred Claim Ratio: {cm['incurred_claim_ratio_pct']}%.")
+        if cm.get("claims_rejected_fy24") is not None:
+            parts.append(f"Claims rejected in FY24: {cm['claims_rejected_fy24']}.")
+        if len(parts) > 1:
+            chunks.append({"sub_id": "metrics", "label": "claim metrics", "text": "\n".join(parts)})
+
+    # --- 2. Aggregator star ratings (Policybazaar, InsuranceDekho, MouthShut) ---
     agg = d.get("aggregator_ratings") or {}
+    rating_parts = [f"AGGREGATOR RATINGS for {name}."]
     for site, info in agg.items():
-        star = (info or {}).get("avg_star")
+        if not isinstance(info, dict):
+            continue
+        star = info.get("avg_star")
         if star is not None:
             count = info.get("review_count")
-            count_part = f" ({count} reviews)" if count else ""
-            parts.append(f"{site.replace('_',' ').title()} rating: {star}/5{count_part}.")
-
-    # Trustpilot
+            count_part = f" from {count} reviews" if count else ""
+            note = info.get("note", "")
+            note_part = f" — {note}" if note else ""
+            rating_parts.append(f"{site.replace('_',' ').title()}: {star}/5{count_part}.{note_part}")
     tp = d.get("trustpilot") or {}
     if tp.get("score") is not None:
-        parts.append(f"Trustpilot: {tp['score']}/5 over {tp.get('review_count','few')} reviews.")
-
-    # Reddit / Youtube sentiment summaries (text fields)
-    for key, label in [
-        ("reddit_sentiment", "Reddit user sentiment"),
-        ("youtube_coverage", "YouTube coverage"),
-        ("in_news", "Recent news"),
-    ]:
-        v = d.get(key)
-        if isinstance(v, dict):
-            summary = v.get("summary") or v.get("note")
-            if summary:
-                parts.append(f"{label}: {summary}")
-        elif isinstance(v, str) and v.strip():
-            parts.append(f"{label}: {v.strip()}")
-
-    # Aggregate score the bot has computed
-    agg_score = d.get("aggregate_score")
-    if isinstance(agg_score, dict):
-        s = agg_score.get("score")
-        rationale = agg_score.get("rationale") or ""
-        if s is not None:
-            parts.append(f"Overall trust score (internal): {s}. {rationale}")
-
-    parts.append(
-        "Use these reviews when a user asks about claim experience, "
-        "service quality, or general reputation. Reviews are dated; "
-        f"last updated {d.get('last_updated','recent')}."
-    )
-    return "\n".join(parts)
+        rating_parts.append(f"Trustpilot: {tp['score']}/5 over {tp.get('review_count','few')} reviews.")
+    if len(rating_parts) > 1:
+        chunks.append({"sub_id": "ratings", "label": "aggregator ratings", "text": "\n".join(rating_parts)})
+
+    # --- 3. Reddit / Quora sentiment + themes ---
+    rs = d.get("reddit_sentiment") or {}
+    if isinstance(rs, dict) and (rs.get("notable_themes") or rs.get("sentiment_overall")):
+        parts = [
+            f"REDDIT AND QUORA USER SENTIMENT for {name}.",
+            f"Overall sentiment: {rs.get('sentiment_overall','mixed')}.",
+            f"Subreddits: {rs.get('subreddit','various')}.",
+            f"Approx mentions last year: {rs.get('mentions_last_year_estimate','few')}.",
+        ]
+        themes = rs.get("notable_themes") or []
+        if themes:
+            parts.append("Notable themes from real user posts:")
+            parts.extend(f"- {t}" for t in themes if isinstance(t, str))
+        chunks.append({"sub_id": "reddit", "label": "reddit sentiment", "text": "\n".join(parts)})
+
+    # --- 4. YouTube creator coverage ---
+    yt = d.get("youtube_coverage") or {}
+    if isinstance(yt, dict) and yt.get("top_creators_who_reviewed"):
+        creators = yt["top_creators_who_reviewed"]
+        parts = [
+            f"YOUTUBE CREATOR REVIEWS of {name}.",
+            f"Overall YouTube sentiment: {yt.get('overall_youtube_sentiment','mixed')}.",
+            "Reviewed by:",
+        ]
+        for c in creators:
+            if isinstance(c, dict):
+                parts.append(f"- {c.get('creator','?')}: \"{c.get('video_title','')}\" — {c.get('video_url','')}")
+        chunks.append({"sub_id": "youtube", "label": "youtube reviews", "text": "\n".join(parts)})
+
+    # --- 5. Recent news items (each a one-liner) ---
+    in_news = d.get("in_news")
+    if isinstance(in_news, list) and in_news:
+        parts = [f"RECENT NEWS about {name} (verified press coverage)."]
+        for item in in_news[:10]:
+            if isinstance(item, dict):
+                hl = item.get("headline","")
+                url = item.get("url","")
+                date = item.get("date","")
+                parts.append(f"- {hl} ({date}) — {url}")
+        if len(parts) > 1:
+            chunks.append({"sub_id": "news", "label": "recent news", "text": "\n".join(parts)})
+
+    # --- 6. Aggregate score + letter grade summary ---
+    agg_score = d.get("aggregate_score") or {}
+    if isinstance(agg_score, dict) and agg_score.get("value_0_100") is not None:
+        parts = [
+            f"OVERALL REPUTATION SUMMARY for {name}.",
+            f"Internal aggregate score: {agg_score.get('value_0_100')}/100 ({agg_score.get('letter_grade','?')}).",
+        ]
+        if agg_score.get("headline"):
+            parts.append(f"Summary: {agg_score['headline']}")
+        if agg_score.get("computation_notes"):
+            parts.append(f"How this score was computed: {agg_score['computation_notes']}")
+        chunks.append({"sub_id": "overall", "label": "overall trust score", "text": "\n".join(parts)})
+
+    return chunks
+
+
+# Backwards-compat shim — keep the old name available so any external
+# callers don't break. Returns the concatenation of all chunks.
+def review_to_paragraph(d: dict) -> str:
+    chunks = review_to_chunks(d)
+    return "\n\n---\n\n".join(c["text"] for c in chunks)
 
 
 def first_verified_url(d: dict) -> str:
@@ -132,7 +188,7 @@ async def main():
     )
     embedder = LocalEmbeddings()
 
-    ok, skipped = 0, 0
+    ok_insurers, ok_chunks, skipped = 0, 0, 0
     for f in files:
         try:
             d = json.load(open(f))
@@ -141,41 +197,46 @@ async def main():
             skipped += 1
             continue
         slug = d.get("insurer_slug") or f.stem
-        chunk_id = f"review_{slug}"
-        text = review_to_paragraph(d)
-        if len(text) < 100:
-            print(f"  SKIP {slug}: rendered text too short ({len(text)} chars)")
+        parent_id = f"review_{slug}"
+        chunks = review_to_chunks(d)
+        if not chunks:
+            print(f"  SKIP {slug}: no embeddable content")
             skipped += 1
             continue
-        [vec] = await embedder.embed([text], input_type="document")
 
-        # Replace any prior chunk for this insurer (idempotent)
+        # Replace any prior chunks for this insurer (idempotent across re-runs)
         try:
-            coll.delete(where={"policy_id": chunk_id})
+            coll.delete(where={"policy_id": parent_id})
         except Exception:
             pass
 
-        coll.add(
-            ids=[chunk_id],
-            documents=[text],
-            embeddings=[vec],
-            metadatas=[{
-                "policy_id":    chunk_id,
+        texts = [c["text"] for c in chunks]
+        vecs = await embedder.embed(texts, input_type="document")
+        url = first_verified_url(d)
+        ids = []
+        metadatas = []
+        for i, c in enumerate(chunks):
+            sub = c["sub_id"]
+            ids.append(f"{parent_id}_{sub}")
+            metadatas.append({
+                "policy_id":    parent_id,                       # share parent — easy to delete-by-insurer
                 "insurer_slug": slug,
                 "policy_name":  f"{d.get('insurer_name', slug)} reviews",
                 "doc_type":     "review",
-                "source_url":   first_verified_url(d),
+                "review_facet": sub,                              # NEW — claim metrics / ratings / reddit / etc.
+                "source_url":   url,
                 "page_start":   0,
                 "page_end":     0,
-                "chunk_idx":    0,
+                "chunk_idx":    i,
                 "local_path":   str(f),
-            }],
-        )
-        print(f"  OK   {slug:22s}  {len(text):4d} chars  -> {chunk_id}")
-        ok += 1
+            })
+        coll.add(ids=ids, documents=texts, embeddings=vecs, metadatas=metadatas)
+        print(f"  OK   {slug:18s}  {len(chunks)} chunks  ({sum(len(t) for t in texts):>5d} chars)")
+        ok_insurers += 1
+        ok_chunks += len(chunks)
 
     print()
-    print(f"Done. Embedded: {ok}, skipped: {skipped}, total: {len(files)}")
+    print(f"Done. Insurers embedded: {ok_insurers}, total chunks: {ok_chunks}, skipped: {skipped}, total files: {len(files)}")
 
 
 if __name__ == "__main__":