Spaces:
Sleeping
Sleeping
| # `rag/` — Retrieval pipeline (corpus → extraction → embeddings → Chroma) | |
| The end-to-end RAG pipeline: download insurer PDFs, parse + chunk, embed with local BGE-small, persist to Chroma, run structured extraction with a Pydantic schema, and serve top-k retrieval at query time. | |
| Lineage for every artefact below is documented in [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md). | |
| ## Build-time scripts | |
| | File | Pipeline stage | What it produces | | |
| | --- | --- | --- | | |
| | `download_corpus.py` | 1. SOURCE → 3. DOWNLOAD | `rag/corpus/<insurer>/*.pdf` + `_manifest.json`. HEAD-checks + magic-byte sniff + retry on 403/timeout. | | |
| | `download_retry.py` | 3. DOWNLOAD | Retries the failures from the previous run. | | |
| | `download_regulatory.py` | 3. DOWNLOAD | IRDAI / regulatory PDFs (deferred from v1; see [ADR-017](../70-docs/60-decisions/ADR-017-irdai-corpus-playwright-rescue.md)). | | |
| | `ingest.py` | 4. PARSE → 5. CHUNK → 6. EMBED → 7. INDEX | The big one. `read_pdf_pages` (pdfplumber) → `chunk_pages` (800-tok / 120 overlap, sentence-aware) → BGE embed → `chromadb.PersistentClient.add(...)`. Carries the in-process HNSW bloat tripwire ([ADR-029](../70-docs/60-decisions/ADR-029-hnsw-bloat-tripwire.md)). | | |
| | `extract.py` | 8. STRUCTURED EXTRACTION | LLM extraction over each PDF using `schema.py::HealthPolicy` (62 fields). `get_brain_llm()` (Brain Main: Gemini 2.5 Flash primary per [ADR-040](../70-docs/60-decisions/ADR-040-google-gemini-primary.md)) — a `NimChainLLM` fallback chain (the separate `get_fast_brain_llm()` accessor was removed in the 2026-05-15 three-chain collapse) across Google → NIM → OpenRouter. Native JSON mode (`response_mime_type=application/json` on Gemini, `response_format={"type":"json_object"}` on NIM). Writes `rag/extracted/<policy_id>.json` + upserts `policies.duckdb`. | | |
| | `build_kb.py` | 9. SCORECARD → KB MIRROR | Runs `backend/scorecard.py` per policy and regenerates the human-readable `kb/policies/<id>.md` tree. | | |
| | `source_map.py` | post-build | Builds `source_map.json` — every chunk → (PDF path, page, span) for citation rendering. | | |
| ## Runtime modules | |
| | File | Stage | Notes | | |
| | --- | --- | --- | | |
| | `retrieve.py` | 10. RETRIEVAL | Top-k Chroma query with policy-id / insurer-slug filters. In-process LRU cache (cap 256) keyed by `(query_norm, top_k, sorted policy_ids, sorted insurer_slugs)`. | | |
| | `schema.py` | 8. STRUCTURED EXTRACTION | The 62-field `HealthPolicy` Pydantic schema — single source of truth for the extracted JSON shape. See `rag/SCHEMA.md` for the field-by-field doc. | | |
| ## Persistent artefacts | |
| | Path | Source of truth | Notes | | |
| | --- | --- | --- | | |
| | `rag/corpus/<insurer>/*.pdf` | insurer CDNs | 206 PDFs (188 product PDFs across 21 insurer slugs + 18 regulatory IRDAI/NHA docs; dedup to 148 marketplace cards). Not in git — hydrated at Docker build from the companion HF dataset. | | |
| | `rag/extracted/<policy_id>.json` | `extract.py` | 201 JSONs, one per policy, conforming to `schema.HealthPolicy`. Generated; never hand-edit. | | |
| | `rag/vectors/chroma.sqlite3` + HNSW binaries | `ingest.py` | Persistent Chroma store. Symlinked to `rag/_hf_dataset_backup/rag/vectors/` for the offline canonical copy. | | |
| | `rag/policies.duckdb` | `extract.py` | DuckDB rollup of the 62-field JSONs; used for SQL-style filters in `backend/main.py`. | | |
| | `rag/source_map.json` | `source_map.py` | chunk_id → (pdf_path, page, span) for the citation links shown in the UI. | | |
| | `rag/SCHEMA.md` | hand-written | Field-by-field documentation of the Pydantic schema. | | |
| ## Subdirectories | |
| - `corpus/` — raw PDFs (one folder per insurer slug). Generated by `download_corpus.py`. Per-PDF folders intentionally do not carry READMEs. | |
| - `extracted/` — 62-field JSONs. Auto-generated by `extract.py`. Do not edit by hand. | |
| - `vectors/` — Chroma persistent store. Treat as opaque — re-build with `python -m rag.ingest` if corrupted. | |
| - `_hf_dataset_backup/rag/{corpus,extracted,vectors}/` — offline canonical mirror of the companion HF Dataset (`rohitsar567/insurance-bot-data`). See [ADR-020](../70-docs/60-decisions/ADR-020-code-data-split-hf-dataset.md). | |
| ## Cold-rebuild | |
| ```bash | |
| python -m rag.download_corpus | |
| python -m rag.download_retry | |
| python -m rag.extract | |
| rm -rf rag/vectors | |
| python -m rag.ingest | |
| python -m rag.build_kb | |
| python -m eval.generate_gold | |
| python -m eval.run | |
| ``` | |
| Total cost from cold: < $2 (BGE local + ~80 LLM extractions). Wall-time: ~30-40 min on a modern laptop. | |
| ## Related | |
| - [ADR-004](../70-docs/60-decisions/ADR-004-hybrid-structured-vector.md) — hybrid structured + vector retrieval rationale | |
| - [ADR-011](../70-docs/60-decisions/ADR-011-bge-local-embeddings.md) — why local BGE replaced Voyage | |
| - [ADR-018](../70-docs/60-decisions/ADR-018-chunk-size-sweep-deferred.md) — 800/120 chunk-size baseline | |
| - [ADR-020](../70-docs/60-decisions/ADR-020-code-data-split-hf-dataset.md) — code-vs-data repo split | |
| - [ADR-029](../70-docs/60-decisions/ADR-029-hnsw-bloat-tripwire.md) — HNSW bloat tripwire (3-layer defence) | |
| - [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) — end-to-end lineage doc | |