# `data/` — Runtime + marketplace data

Three classes of file live here, intentionally side-by-side:

1. **Runtime state** — written by the live server during normal operation (`profiles/`, `sessions/`, `llm_health.json`, `llm_usage.jsonl`).
2. **Pre-computed marketplace data** — curated artefacts the server reads on every relevant turn (`policy_facts/`, `premiums/`, `reviews/`).
3. **Source/lineage maps** — human-readable manifests of where every claim traces back to (`corpus_urls.md`, `regulatory_urls.md`, `information_source_map.md`).

The structured policy schema and PDFs themselves live under `rag/`. This folder is downstream.

## Top-level files

| File | What it is | Owner |
| --- | --- | --- |
| `corpus_urls.md` | Discovery manifest — every PDF URL ingested into `rag/corpus/`. | discovery agent / `tools/check_link_rot.py` |
| `regulatory_urls.md` | IRDAI / regulatory PDF URLs. See [ADR-017](../70-docs/60-decisions/ADR-017-irdai-corpus-playwright-rescue.md). | discovery agent |
| `information_source_map.md` | Human-readable claim → URL → verdict map. Master audit doc for the Source Methodology directive. Mirror of `eval/info_source_map.json`. | `tools/info_source_map.py` |
| `llm_health.json` | Last per-provider health-probe snapshot (latency, success, last error). Powers the admin tab. | `backend/llm_health.py` |
| `llm_usage.jsonl` | Append-only per-call log: provider, model, tokens, latency, success. Aggregated in the admin tab. | `backend/main.py` |

## Subdirectories

| Path | Class | Contents |
| --- | --- | --- |
| `profiles/` | runtime | Persistent named-profile JSON store (KI-040). One file per user, normalised-name slug. See `data/profiles/README.md`. |
| `sessions/` | runtime | Per-session conversation state JSONs. Ephemeral — pruned periodically. Currently includes `anonymous.json` (no-name fallback). |
| `policy_facts/` | pre-computed | **256 curated JSONs**, one per policy variant. Each field carries `{value, unit?, source_pdf_path, source_quote}` provenance. The Indian-BFSI-audit-grade machine source; `kb/policies/*.md` are the human-readable mirror. See `_curation_report.md` for the three batches that built it. |
| `policies/` | pre-computed | Subfolder per insurer with PDFs / supplementary text used for one-off lookups outside the main ingest pipeline. |
| `premiums/` | pre-computed | `illustrative_premiums.json` — sample starting premiums pulled from PolicyBazaar / JoinDitto / Beshak + insurer rate cards (2026-05-13). Refreshed by `tools/refresh_premiums.py`. **Illustrative only** per [ADR-007](../70-docs/60-decisions/ADR-007-illustrative-pricing.md). |
| `reviews/` | pre-computed | One JSON per insurer with IRDAI claim-settlement metrics, complaints/10K, aggregator sentiment, news tone. Index + leaderboard in `reviews/INDEX.md`. Source: IRDAI Annual Report 2023-24. |

## Provenance + KPIs

| Metric | Value (2026-05-14) | Where to verify |
| --- | --- | --- |
| Curated policy variants | 256 | `data/policy_facts/` file count |
| Per-policy avg field completeness | 83.5% (Batch 1) | `data/policy_facts/_curation_report.md` |
| Information-source-map verdicts | ✅ 798 · ⚠️ 321 · ❌ 0 · ⏳ 1385 | `eval/info_source_map.json` |

## Related

- [`kb/AUDIT_TRAIL.md`](../kb/AUDIT_TRAIL.md) — end-to-end lineage; `data/policy_facts/` is stage 8 output
- [`kb/INDEX.md`](../kb/INDEX.md) — policy index with completeness % per file
- [ADR-007](../70-docs/60-decisions/ADR-007-illustrative-pricing.md) — pricing is illustrative, never a real quote
- [ADR-009](../70-docs/60-decisions/ADR-009-19-insurer-comprehensive-schema.md) — 19-insurer scope + 48-field schema