Deploy: Stack A (NIM brain + Maverick judge + Sarvam voice). D-019.

README.md

@@ -72,3 +72,14 @@ See [`docs/01-requirements.md` §7](docs/01-requirements.md).
 - **UI**: Streamlit
 
 Each pick is justified in [`docs/decisions.md`](docs/decisions.md).
+
+## Deploying changes
+
+All extraction, embedding, and Chroma writes for the curated corpus run on the developer's Mac. The deployed HF Space serves pre-built indexes and does not auto-ingest (the one exception is user-uploaded PDFs, which embed on-demand into an isolated `user_uploads_quarantine` collection). Full rationale and the per-stage where-it-runs table: [`docs/ingestion_policy.md`](docs/ingestion_policy.md).
+
+After any corpus change, run these four commands sequentially from the project root with the local `.venv` active:
+
+- [ ] `.venv/bin/python -m rag.ingest` — rebuild the Chroma index locally
+- [ ] `.venv/bin/python tools/upload_vectors_to_dataset.py` — push `rag/vectors/` to the HF dataset
+- [ ] `.venv/bin/python tools/upload_extracted_to_dataset.py` — push `rag/extracted/` (only if structured JSONs changed)
+- [ ] `.venv/bin/python tools/upload_to_hf.py` — push code to the Space (triggers a Docker rebuild that pulls the dataset)

backend/providers/local_embeddings.py

@@ -27,11 +27,39 @@ class LocalEmbeddings(EmbeddingsProvider):
         device: Optional[str] = None,
     ):
         # Lazy import so this module loads fast even if model isn't downloaded
+        import os
         from sentence_transformers import SentenceTransformer
 
+        # Device autodetect: MPS on Apple Silicon when available (2-3x faster
+        # than CPU on long chunks), CUDA if present, else CPU. Honor explicit
+        # override via constructor arg OR EMBED_DEVICE env var so HF Space
+        # (no MPS) and local Mac (with MPS) pick the right path.
+        if device is None:
+            device = os.environ.get("EMBED_DEVICE", "").strip() or None
+        if device is None:
+            try:
+                import torch
+                if torch.backends.mps.is_available():
+                    device = "mps"
+                elif torch.cuda.is_available():
+                    device = "cuda"
+                else:
+                    device = "cpu"
+            except Exception:
+                device = "cpu"
+
         self.model_name = model_name
+        self.device = device
         self.model = SentenceTransformer(model_name, device=device)
         self.dimension = self.model.get_sentence_embedding_dimension()
+        # Warm-up call on MPS — first kernel JIT compile is ~3-5s; doing it
+        # in __init__ rather than first encode() makes the first user request
+        # fast. CPU/CUDA skip this (their first call has no JIT penalty).
+        if device == "mps":
+            try:
+                self.model.encode(["warmup"] * 2, batch_size=2, show_progress_bar=False)
+            except Exception:
+                pass
 
     async def embed(
         self,
@@ -43,10 +71,13 @@ class LocalEmbeddings(EmbeddingsProvider):
         # BGE recommends a small query-side instruction; not strictly required
         if input_type == "query":
             texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts]
-        # Encode synchronously on CPU/GPU; small batches don't need true async
+        # Batch size scales by device: MPS / CUDA throughput benefits from
+        # bigger batches; CPU prefers smaller to avoid memory pressure on M1.
+        # 800-token chunks at batch_size=64 is ~50 MB which fits 8GB M1 fine.
+        batch = 64 if self.device in ("mps", "cuda") else 32
         vectors = self.model.encode(
             texts,
-            batch_size=32,
+            batch_size=batch,
             show_progress_bar=False,
             convert_to_numpy=True,
             normalize_embeddings=True,

docs/ingestion_policy.md

@@ -0,0 +1,99 @@
+# Ingestion Policy (2026-05-14)
+
+**Headline:** All extraction, embedding, and Chroma writes for the curated
+policy corpus run on the developer's Mac. The deployed Hugging Face Space
+serves pre-built indexes only. The single exception is user-uploaded PDFs,
+which are embedded on-demand into a SEPARATE `user_uploads_quarantine`
+Chroma collection isolated from the main `policies` corpus.
+
+---
+
+## Where things run
+
+| Pipeline stage                              | Runs on Mac (`.venv`) | Runs in Space (Docker) |
+|---------------------------------------------|:---------------------:|:----------------------:|
+| Source-PDF download (`rag/corpus/**`)       | yes                   | no                     |
+| Text extraction (pdfplumber → JSON)         | yes                   | no                     |
+| Chunking (800-token windows, 120 overlap)   | yes                   | no                     |
+| Embedding (BAAI/bge-small-en-v1.5)          | yes                   | **only quarantine**    |
+| Chroma write — `policies` collection        | yes                   | no                     |
+| Chroma write — `user_uploads_quarantine`    | no                    | yes (on-demand)        |
+| HF dataset push (`rag/vectors/**`, `rag/extracted/**`, `rag/corpus/**`) | yes | no |
+| HF Space push (code only)                   | yes                   | no                     |
+| Serving (retrieval, LLM, voice, frontend)   | no                    | yes                    |
+
+Reading the column "Runs in Space": the only embedding work the deployed
+container ever performs is on-demand quarantine embedding of a user-uploaded
+PDF inside `POST /api/upload-policy`. Every other embedding has been baked
+into the image at Docker build time via `huggingface_hub.snapshot_download`
+of the companion dataset (`rohitsar567/insurance-bot-data`).
+
+## Local workflow
+
+After any change to the curated corpus, run these on the Mac in order:
+
+```bash
+# 1. Re-build the Chroma index against the local PDFs + extracted JSONs.
+.venv/bin/python -m rag.ingest
+
+# 2. Push the freshly-built rag/vectors/ to the HF dataset so the next
+#    Space rebuild pulls a ready-to-serve index.
+.venv/bin/python tools/upload_vectors_to_dataset.py
+
+# 3. Only if structured extraction JSONs changed (rag/extracted/*.json) —
+#    e.g. you re-ran an NIM extraction batch.
+.venv/bin/python tools/upload_extracted_to_dataset.py
+
+# 4. Push the code to the HF Space. This triggers a Docker rebuild; the
+#    build pulls rag/vectors/ + rag/extracted/ + rag/corpus/ from the
+#    dataset (see Dockerfile `allow_patterns`).
+.venv/bin/python tools/upload_to_hf.py
+```
+
+The four steps are sequential — step 4 must come last because the Space
+rebuild snapshots whatever state the dataset is in at build time. If you
+push code before the vectors are synced, the Space will boot against a
+stale index (or, if the schema changed, fail the entrypoint validation).
+
+## Why
+
+Prior policy auto-ingested on Space boot if Chroma looked empty or broken.
+That created two bad failure modes:
+
+1. **Confusing schema breakage with normal boot.** A breaking Chroma schema
+   change (e.g. a chromadb version bump) silently triggered a 20+ minute
+   re-ingest during `APP_STARTING`. The Space logged nothing user-visible
+   while it churned, then either succeeded (slow, but fine) or failed deep
+   inside the embedder. Operators could not tell which.
+2. **Resource cost on the wrong machine.** Free-tier Spaces have a CPU
+   budget and a 1 GB image cap. Embedding ~190 PDFs on Space CPU was
+   slower than on a developer Mac and pushed boot far past the platform's
+   health-check window.
+
+Fail-fast is better: `entrypoint.sh` now validates Chroma is readable and
+populated, and exits with a loud error if not. The fix is documented at
+exit time (run `rag.ingest` locally, push vectors, redeploy). Total
+boot-to-serving time on the Space is now seconds, not tens of minutes.
+
+## Exception — user uploads
+
+`POST /api/upload-policy` accepts an arbitrary PDF from the public web and
+must embed it before the chatbot can answer questions about it. To keep
+this off the main corpus while still allowing the feature:
+
+- The endpoint writes into a SEPARATE Chroma collection named
+  `user_uploads_quarantine` (created lazily via
+  `rag.ingest.get_quarantine_collection`).
+- Every chunk in that collection is tagged with the uploading session's
+  `session_id`. Retrieval against the quarantine collection is scoped to
+  that `session_id`, so one user's upload never surfaces in another
+  user's session — and never surfaces in queries against the main
+  `policies` collection at all.
+- The quarantine collection is the ONLY place the deployed Space ever
+  writes embeddings. It does not back-propagate to the dataset, does not
+  persist across Space rebuilds (it lives only in the Space's working
+  Chroma directory), and is not part of any evaluation set.
+
+If a user-uploaded PDF turns out to be worth curating, the operator pulls
+it into `rag/corpus/<insurer>/` on the Mac and re-runs the local workflow
+above. There is no in-place promotion from quarantine to `policies`.

entrypoint.sh

@@ -1,9 +1,16 @@
 #!/bin/sh
-# Container entrypoint:
-#   1. If the Chroma vector store is empty, run ingestion from rag/corpus/
-#      (one-time ~5 min on first boot; cached on HF persistent disk for
-#      subsequent boots)
-#   2. Start uvicorn
+# Container entrypoint (2026-05-14 policy update):
+#   1. Validate Chroma is readable + populated.
+#   2. If empty/broken: FAIL FAST with a loud error — DO NOT auto-ingest.
+#      Ingestion runs on the developer's local Mac (faster CPU, visible
+#      progress). The deployed Space serves pre-built indexes only.
+#      The single exception is the user_uploads_quarantine collection,
+#      which /api/upload-policy embeds on-demand per uploading session.
+#   3. Start uvicorn.
+#
+# Why: previously the Space silently re-ingested for 20+ min during APP_STARTING
+# (output piped through `tail -30` so nothing visible), making schema breakage
+# look identical to "still booting". Fail-fast surfaces ingest gaps immediately.
 
 set -e
 
@@ -46,9 +53,16 @@ except Exception as e:
     print(f'[entrypoint] Chroma load FAILED: {type(e).__name__}: {e}')
     sys.exit(1)
 " || (
-    echo "[entrypoint] Wiping vector store and re-ingesting from corpus..."
-    rm -rf /app/rag/vectors/* 2>/dev/null || true
-    python -m rag.ingest 2>&1 | tail -30
+    echo "[entrypoint] ============================================================"
+    echo "[entrypoint] FATAL: Chroma vector store is empty or schema-incompatible."
+    echo "[entrypoint] Auto-ingest is DISABLED (2026-05-14 policy)."
+    echo "[entrypoint]"
+    echo "[entrypoint] Fix on the developer Mac:"
+    echo "[entrypoint]   .venv/bin/python -m rag.ingest"
+    echo "[entrypoint]   .venv/bin/python tools/upload_extracted_to_dataset.py"
+    echo "[entrypoint]   # plus sync rag/vectors/ to the dataset, then redeploy"
+    echo "[entrypoint] ============================================================"
+    exit 1
 )
 
 # Start the server

rag/retrieve.py

@@ -68,6 +68,24 @@ def _is_regulatory_intent(query: str) -> bool:
     return bool(_REGULATORY_TRIGGERS.search(query or ""))
 
 
+# Review-intent triggers: when a query is about insurer reputation /
+# claim experience / customer satisfaction, surface the review chunks
+# (ingested via tools/ingest_reviews.py, doc_type='review').
+_REVIEW_TRIGGERS = _re.compile(
+    r"\b(review|reviews|rating|ratings|reputation|complaint|complaints|"
+    r"trustpilot|policybazaar|insurancedekho|joinditto|claim experience|"
+    r"claim settlement|claim ratio|complaint ratio|customer service|"
+    r"good service|bad service|user experience|reddit|youtube|"
+    r"feedback|testimonial|sentiment|trust|reliable|reliability)\b",
+    flags=_re.IGNORECASE,
+)
+REVIEW_BOOST = 1.15  # slight boost so reviews appear in reputation Qs
+
+
+def _is_review_intent(query: str) -> bool:
+    return bool(_REVIEW_TRIGGERS.search(query or ""))
+
+
 def _build_chunk(cid: str, doc: str, meta: dict, score: float) -> RetrievedChunk:
     return RetrievedChunk(
         chunk_id=cid,
@@ -215,6 +233,42 @@ async def retrieve(
             # Regulatory boost is additive; failure shouldn't kill the main result
             pass
 
+    # Review boost pass — when query asks about insurer reputation / claim
+    # experience / service quality. Filters to the relevant insurer if
+    # insurer_slugs is set; otherwise queries across all reviews.
+    if _is_review_intent(query):
+        try:
+            where_review: dict = {"doc_type": "review"}
+            if insurer_slugs:
+                # Combine doc_type + insurer filter
+                where_review = {
+                    "$and": [
+                        {"doc_type": "review"},
+                        {"insurer_slug": {"$in": insurer_slugs}},
+                    ]
+                }
+            rev_res = collection.query(
+                query_embeddings=[query_vec],
+                n_results=3,
+                where=where_review,
+            )
+            if rev_res["ids"] and rev_res["ids"][0]:
+                seen = {c.chunk_id for c in out}
+                rev_chunks: list[RetrievedChunk] = []
+                for cid, doc, meta, dist in zip(
+                    rev_res["ids"][0], rev_res["documents"][0],
+                    rev_res["metadatas"][0], rev_res["distances"][0],
+                ):
+                    if cid in seen:
+                        continue
+                    boosted = (1.0 - dist) * REVIEW_BOOST
+                    rev_chunks.append(_build_chunk(cid, doc, meta, boosted))
+                merged = sorted(out + rev_chunks, key=lambda c: c.score, reverse=True)
+                out = merged[:top_k]
+        except Exception:
+            # Review boost is additive; failure shouldn't kill the main result
+            pass
+
     return out
 
 

requirements.txt

@@ -15,7 +15,11 @@ requests==2.34.0
 pdfplumber==0.11.4
 
 # RAG: vector store + structured store
-chromadb==0.5.20
+# 2026-05-14: pinned to 1.5.9 to match the local .venv version that builds
+# rag/vectors. Schema drift between chroma versions caused KeyError('_type')
+# in earlier deploys, wiping the index and forcing a 45-min re-ingest.
+# Bump both local AND this pin together if upgrading.
+chromadb==1.5.9
 duckdb==1.1.3
 
 # Local embeddings (BGE small) — replaces Voyage in v1 to bypass free-tier rate limits

tools/ingest_reviews.py

@@ -0,0 +1,182 @@
+"""Ingest insurer reviews into the main Chroma `policies` collection.
+
+For each insurer review JSON in `data/reviews/`:
+  1. Render the structured review into a natural-language paragraph that
+     captures the gist of an insurer's reputation: claim settlement %,
+     complaint rate, aggregator ratings, sentiment summary, news flags.
+  2. Embed via LocalEmbeddings (BGE-small).
+  3. Write to the same Chroma `policies` collection with
+       insurer_slug = <slug>
+       policy_id    = "review_<slug>"
+       doc_type     = "review"
+       source_url   = first verified review URL we have
+  4. Idempotent: re-running replaces the existing chunk for that insurer.
+
+After ingest, `retrieve()` will surface these chunks for queries like
+"is HDFC ERGO's claim experience good?" or "what do users say about
+Care Health?" — semantic recall over reviews, citable back to source.
+
+Run AFTER the main rag.ingest finishes (avoids Chroma write contention).
+  .venv/bin/python tools/ingest_reviews.py
+"""
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+
+import chromadb
+from chromadb.config import Settings
+
+from backend.config import settings
+from backend.providers.local_embeddings import LocalEmbeddings
+
+
+ROOT = Path(__file__).resolve().parent.parent
+REVIEWS_DIR = ROOT / "data" / "reviews"
+
+
+def review_to_paragraph(d: dict) -> str:
+    """Render a structured review JSON into a single English paragraph
+    suitable for embedding + retrieval."""
+    parts: list[str] = []
+    name = d.get("insurer_name") or d.get("insurer_slug")
+    parts.append(f"USER REVIEWS AND REPUTATION — {name}.")
+
+    # Hard claim metrics first — most-cited numbers
+    cm = d.get("claim_metrics") or {}
+    if cm.get("claim_settlement_ratio_pct") is not None:
+        parts.append(
+            f"Claim Settlement Ratio: {cm['claim_settlement_ratio_pct']}% "
+            f"({cm.get('claim_settlement_ratio_year','recent')}, per IRDAI)."
+        )
+    if cm.get("complaints_per_10k_policies") is not None:
+        parts.append(
+            f"Complaints per 10,000 policies: {cm['complaints_per_10k_policies']} "
+            f"({cm.get('complaints_year','recent')})."
+        )
+    if cm.get("incurred_claim_ratio_pct") is not None:
+        parts.append(f"Incurred Claim Ratio: {cm['incurred_claim_ratio_pct']}%.")
+
+    # Aggregator star ratings (Policybazaar, InsuranceDekho, Ditto, etc.)
+    agg = d.get("aggregator_ratings") or {}
+    for site, info in agg.items():
+        star = (info or {}).get("avg_star")
+        if star is not None:
+            count = info.get("review_count")
+            count_part = f" ({count} reviews)" if count else ""
+            parts.append(f"{site.replace('_',' ').title()} rating: {star}/5{count_part}.")
+
+    # Trustpilot
+    tp = d.get("trustpilot") or {}
+    if tp.get("score") is not None:
+        parts.append(f"Trustpilot: {tp['score']}/5 over {tp.get('review_count','few')} reviews.")
+
+    # Reddit / Youtube sentiment summaries (text fields)
+    for key, label in [
+        ("reddit_sentiment", "Reddit user sentiment"),
+        ("youtube_coverage", "YouTube coverage"),
+        ("in_news", "Recent news"),
+    ]:
+        v = d.get(key)
+        if isinstance(v, dict):
+            summary = v.get("summary") or v.get("note")
+            if summary:
+                parts.append(f"{label}: {summary}")
+        elif isinstance(v, str) and v.strip():
+            parts.append(f"{label}: {v.strip()}")
+
+    # Aggregate score the bot has computed
+    agg_score = d.get("aggregate_score")
+    if isinstance(agg_score, dict):
+        s = agg_score.get("score")
+        rationale = agg_score.get("rationale") or ""
+        if s is not None:
+            parts.append(f"Overall trust score (internal): {s}. {rationale}")
+
+    parts.append(
+        "Use these reviews when a user asks about claim experience, "
+        "service quality, or general reputation. Reviews are dated; "
+        f"last updated {d.get('last_updated','recent')}."
+    )
+    return "\n".join(parts)
+
+
+def first_verified_url(d: dict) -> str:
+    """Pick a single canonical URL to attach as source_url on the chunk.
+    Prefer IRDAI claim-stats page, then Policybazaar, then company."""
+    cm = d.get("claim_metrics") or {}
+    for k in ("source_irdai_url", "source_secondary_url", "source_company_url"):
+        if cm.get(k):
+            return cm[k]
+    agg = d.get("aggregator_ratings") or {}
+    for site_info in agg.values():
+        if isinstance(site_info, dict) and site_info.get("url"):
+            return site_info["url"]
+    return ""
+
+
+async def main():
+    files = sorted(REVIEWS_DIR.glob("*.json"))
+    if not files:
+        print(f"No review JSONs found in {REVIEWS_DIR}")
+        return
+
+    client = chromadb.PersistentClient(
+        path=str(settings.VECTORS_DIR),
+        settings=Settings(anonymized_telemetry=False),
+    )
+    coll = client.get_or_create_collection(
+        name="policies",
+        metadata={"hnsw:space": "cosine"},
+    )
+    embedder = LocalEmbeddings()
+
+    ok, skipped = 0, 0
+    for f in files:
+        try:
+            d = json.load(open(f))
+        except Exception as e:
+            print(f"  SKIP {f.name}: {type(e).__name__}: {e}")
+            skipped += 1
+            continue
+        slug = d.get("insurer_slug") or f.stem
+        chunk_id = f"review_{slug}"
+        text = review_to_paragraph(d)
+        if len(text) < 100:
+            print(f"  SKIP {slug}: rendered text too short ({len(text)} chars)")
+            skipped += 1
+            continue
+        [vec] = await embedder.embed([text], input_type="document")
+
+        # Replace any prior chunk for this insurer (idempotent)
+        try:
+            coll.delete(where={"policy_id": chunk_id})
+        except Exception:
+            pass
+
+        coll.add(
+            ids=[chunk_id],
+            documents=[text],
+            embeddings=[vec],
+            metadatas=[{
+                "policy_id":    chunk_id,
+                "insurer_slug": slug,
+                "policy_name":  f"{d.get('insurer_name', slug)} reviews",
+                "doc_type":     "review",
+                "source_url":   first_verified_url(d),
+                "page_start":   0,
+                "page_end":     0,
+                "chunk_idx":    0,
+                "local_path":   str(f),
+            }],
+        )
+        print(f"  OK   {slug:22s}  {len(text):4d} chars  -> {chunk_id}")
+        ok += 1
+
+    print()
+    print(f"Done. Embedded: {ok}, skipped: {skipped}, total: {len(files)}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

tools/set_hf_secrets.py

@@ -23,10 +23,18 @@ ROOT = Path(__file__).resolve().parent.parent
 load_dotenv(ROOT / ".env")
 
 REPO_ID = "rohitsar567/InsuranceBot"
-# Active secrets (D-019 Stack A): only these are read by the running code.
-SECRETS_TO_SET = ["SARVAM_API_KEY", "VOYAGE_API_KEY", "NVIDIA_NIM_API_KEY"]
-# Legacy secrets to delete (retired providers — see D-019).
-SECRETS_TO_DELETE = ["GROQ_API_KEY", "OPENROUTER_API_KEY", "CEREBRAS_API_KEY", "DEEPSEEK_API_KEY"]
+# Active secrets read by the running code (post-D-022 + admin panel):
+#   - SARVAM / NIM = the live providers
+#   - VOYAGE       = legacy (kept for back-compat with old extracted artifacts)
+#   - OPENROUTER / GROQ = optional cross-provider fallbacks (re-added 2026-05-14)
+#   - ADMIN_*      = control-panel gate (IP allowlist + password)
+SECRETS_TO_SET = [
+    "SARVAM_API_KEY", "VOYAGE_API_KEY", "NVIDIA_NIM_API_KEY",
+    "OPENROUTER_API_KEY", "GROQ_API_KEY",
+    "ADMIN_IP_ALLOWLIST", "ADMIN_PASSWORD",
+]
+# Truly retired providers — delete from Space to prevent confusion.
+SECRETS_TO_DELETE = ["CEREBRAS_API_KEY", "DEEPSEEK_API_KEY"]
 
 
 def main():

tools/upload_all_to_dataset.py

@@ -0,0 +1,44 @@
+"""One-command sync: extracted + corpus + vectors -> insurance-bot-data dataset.
+
+Runs the three companion uploaders sequentially. Use this after a full local
+pipeline run on the developer Mac (extract -> corpus refresh -> rag.ingest).
+
+  .venv/bin/python tools/upload_all_to_dataset.py
+
+Each step exits non-zero on failure and short-circuits the rest, so a partial
+sync never silently happens. The Space's Dockerfile snapshot_downloads the
+whole dataset at build time, so all three folders need to be in sync before
+running `tools/upload_to_hf.py`.
+"""
+from __future__ import annotations
+import sys
+
+from tools import (
+    upload_extracted_to_dataset,
+    upload_corpus_to_dataset,
+    upload_vectors_to_dataset,
+)
+
+
+STEPS = [
+    ("extracted JSONs", upload_extracted_to_dataset.main),
+    ("corpus PDFs",     upload_corpus_to_dataset.main),
+    ("Chroma vectors",  upload_vectors_to_dataset.main),
+]
+
+
+def main() -> int:
+    n = len(STEPS)
+    for i, (label, fn) in enumerate(STEPS, start=1):
+        print(f"\n=== [{i}/{n}] {label} ===")
+        rc = fn() or 0
+        if rc != 0:
+            print(f"\nABORT: step {i}/{n} ({label}) returned exit {rc}. "
+                  "Fix and re-run; later steps were NOT executed.")
+            return rc
+    print(f"\nAll {n} dataset folders synced. Next: `tools/upload_to_hf.py` to deploy the Space.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/upload_corpus_to_dataset.py

@@ -0,0 +1,44 @@
+"""Sync rag/corpus/ (raw policy PDFs) to insurance-bot-data HF dataset.
+
+Run after any corpus refresh (new PDFs added or refreshed via
+rag/download_corpus.py). The Space's Dockerfile pulls rag/corpus/** at build
+time so the deployed container has the source PDFs available for citation +
+ingestion pipelines.
+"""
+from __future__ import annotations
+import os
+from pathlib import Path
+from dotenv import load_dotenv
+from huggingface_hub import HfApi
+
+ROOT = Path(__file__).resolve().parent.parent
+load_dotenv(ROOT / ".env")
+
+def main():
+    api = HfApi(token=os.environ["HF_TOKEN"])
+    corpus_dir = ROOT / "rag" / "corpus"
+    if not corpus_dir.exists() or not any(corpus_dir.iterdir()):
+        print("ERROR: rag/corpus/ is empty. Run `rag/download_corpus.py` first.")
+        return 1
+    # Print size + file count before upload
+    total = 0
+    nfiles = 0
+    for p in corpus_dir.rglob("*"):
+        if p.is_file():
+            total += p.stat().st_size
+            nfiles += 1
+    print(f"Syncing rag/corpus/ ({total/1024/1024:.1f} MB, {nfiles} files) to insurance-bot-data ...")
+    api.upload_folder(
+        folder_path=str(corpus_dir),
+        path_in_repo="rag/corpus",
+        repo_id="rohitsar567/insurance-bot-data",
+        repo_type="dataset",
+        commit_message="Sync rag/corpus/ — raw policy PDFs (Mac local refresh)",
+        ignore_patterns=["**/.DS_Store", "*.tmp"],
+    )
+    print("Done.")
+    return 0
+
+if __name__ == "__main__":
+    import sys
+    sys.exit(main())

tools/upload_vectors_to_dataset.py

@@ -0,0 +1,44 @@
+"""Sync rag/vectors/ (pre-built Chroma index) to insurance-bot-data HF dataset.
+
+Run AFTER `.venv/bin/python -m rag.ingest` on the developer Mac. The Space's
+Dockerfile pulls rag/vectors/** at build time so the deployed container has a
+ready-to-serve index. As of 2026-05-14 the Space no longer auto-ingests on
+boot — see entrypoint.sh.
+"""
+from __future__ import annotations
+import os
+from pathlib import Path
+from dotenv import load_dotenv
+from huggingface_hub import HfApi
+
+ROOT = Path(__file__).resolve().parent.parent
+load_dotenv(ROOT / ".env")
+
+def main():
+    api = HfApi(token=os.environ["HF_TOKEN"])
+    vectors_dir = ROOT / "rag" / "vectors"
+    if not vectors_dir.exists() or not any(vectors_dir.iterdir()):
+        print("ERROR: rag/vectors/ is empty. Run `.venv/bin/python -m rag.ingest` first.")
+        return 1
+    # Print size + file count before upload
+    total = 0
+    nfiles = 0
+    for p in vectors_dir.rglob("*"):
+        if p.is_file():
+            total += p.stat().st_size
+            nfiles += 1
+    print(f"Syncing rag/vectors/ ({total/1024/1024:.1f} MB, {nfiles} files) to insurance-bot-data ...")
+    api.upload_folder(
+        folder_path=str(vectors_dir),
+        path_in_repo="rag/vectors",
+        repo_id="rohitsar567/insurance-bot-data",
+        repo_type="dataset",
+        commit_message="Sync rag/vectors/ — pre-built Chroma index (Mac local ingest)",
+        ignore_patterns=["*.tmp", "**/.DS_Store"],
+    )
+    print("Done. Now run `tools/upload_to_hf.py` to deploy Space.")
+    return 0
+
+if __name__ == "__main__":
+    import sys
+    sys.exit(main())