rag/SCHEMA.md

HealthPolicy Schema — design notes

Canonical record for one Indian health insurance policy variant. One row = one (insurer, policy_name, variant) tuple. Schema lives in rag/schema.py.

The shape is grounded in the IRDAI Customer Information Sheet (CIS) — the regulator-mandated one-page disclosure every insurer must publish for every product — supplemented by the comparison dimensions used by PolicyBazaar / InsuranceDekho / Acko so that extracted records support both regulator-grade and consumer-grade filtering.

Field groupings

#	Group	Fields	Purpose
1	Identity & metadata	policy_id, insurer_name, insurer_slug, policy_name, policy_type, uin_code	Primary key + IRDAI cross-reference.
2	Eligibility	min_entry_age_years, max_entry_age_years, max_renewal_age_years, min_child_entry_age_days, family_composition_allowed, residency_requirement	First-cut filter: "can this person even buy it?"
3	Sum insured & premium	sum_insured_options_inr, premium_payment_modes, premium_range_indicative_inr, premium_payment_term_years, grace_period_days, free_look_period_days	Affordability + billing flexibility.
4	Waiting periods	initial_waiting_period_days, pre_existing_disease_waiting_months, specific_disease_waiting_months, specific_diseases_listed, maternity_waiting_months, sub_limits_waiting_notes	The single biggest source of claim disputes — buyer must understand these before signing.
5	Coverage scope	inpatient_hospitalization, pre/post_hospitalization_days, day_care_treatments, domiciliary_treatment, ayush_coverage, maternity_coverage, newborn_coverage, organ_donor_expenses, ambulance_cover, critical_illness_cover, restoration_benefit, no_claim_bonus_pct, no_claim_bonus_cap_pct, preventive_health_checkup	What's covered. Each benefit uses the reusable CoverageItem shape (covered, limit_inr, limit_text, notes) so the verbatim CIS wording stays available for citation.
6	Sub-limits & caps	room_rent_capping, icu_capping, copayment_pct, copayment_trigger_notes, disease_wise_sub_limits, deductible_amount_inr	What's not fully covered — the hidden gotchas.
7	Geography & network	geographic_coverage, worldwide_emergency_cover, network_hospital_count, cashless_treatment_supported	Where the policy works.
8	Exclusions	permanent_exclusions, temporary_exclusions, notable_exclusions_summary	IRDAI standardised the permanent-exclusion list in 2020 — relatively easy to extract.
9	Claim & service	claim_settlement_ratio_pct, claim_process_summary, tat_cashless_authorization_hours	Trust signals.
10	Riders	available_riders, top_rider_examples, rider_premium_indicative_inr	Up-sell surface.
11	Source metadata	source_pdf_path, source_pdf_url, last_updated_date, extraction_confidence_pct	Provenance & quality gating.

Field count: ~48, mostly Optional[...] because PDF extraction is lossy.

Critical vs nice-to-have

Critical for side-by-side comparison — these drive almost every buyer decision and must be extracted reliably:

• sum_insured_options_inr, policy_type
• pre_existing_disease_waiting_months, initial_waiting_period_days, specific_disease_waiting_months, maternity_waiting_months
• room_rent_capping, icu_capping, copayment_pct, deductible_amount_inr
• pre_hospitalization_days, post_hospitalization_days
• no_claim_bonus_pct, restoration_benefit
• ayush_coverage, maternity_coverage, critical_illness_cover
• network_hospital_count, cashless_treatment_supported

Nice-to-have — useful for narrative / pitch but not deal-breakers:

• top_rider_examples, rider_premium_indicative_inr
• preventive_health_checkup, domiciliary_treatment
• worldwide_emergency_cover
• disease_wise_sub_limits (the dict can stay sparse)

Likely-hard-to-extract fields (and why)

Field	Why it's hard	Where to actually get it
claim_settlement_ratio_pct	Not in the policy wordings PDF at all. Insurer-level, not policy-level.	IRDAI Annual Report (Statement 11) — separate scrape, joined on insurer_slug.
network_hospital_count	Quoted in marketing pages, rarely in wordings. Changes weekly.	Insurer's hospital-locator API or the IRDAI "Network Hospital" portal.
premium_range_indicative_inr	Wordings never contain pricing.	Public quote engines (PolicyBazaar etc.) for a fixed benchmark profile.
disease_wise_sub_limits	Usually buried in an annexure with inconsistent table layouts.	Targeted second-pass extraction with table-aware models (Camelot / pdfplumber).
tat_cashless_authorization_hours	IRDAI mandated 1 hour in 2024, but older PDFs still say "as per regulations".	Default to 1.0 if absent and policy is post-2024; flag otherwise.
uin_code	Present but easy to confuse with similar product codes; same insurer reuses prefixes.	Regex [A-Z]{4,6}HLIP\d{5}V\d{6} with cross-check against IRDAI's product master.
specific_diseases_listed	Listed in an annexure, often with sub-bullets; needs structure-preserving extraction.	LLM extraction with a clear schema example shot.

extraction_confidence_pct is the gating signal — records below ~70 should be flagged for human review before being served to users.

v2 expansion: Life / Motor / Travel

The schema is forward-compatible without breaking changes:

1. Shared header. policy_id, insurer_name, insurer_slug, policy_name, policy_type, uin_code, plus the entire Source metadata group, apply to every line of business. Move them into a shared PolicyBase mixin when the second LOB lands.
2. Sibling models. Create LifePolicy, MotorPolicy, TravelPolicy alongside HealthPolicy. Each inherits the shared header and adds its own category-specific groups (e.g. LifePolicy adds policy_term_years, death_benefit_inr, maturity_benefit_inr, surrender_value_table).
3. Discriminator field. Add line_of_business: Literal["health","life","motor","travel"] at the base. Storage and retrieval layers route by this field.
4. Backward compatibility. HealthPolicy keeps Config.extra = "allow", so any v2 keys that briefly leak into a health record (during migration) are preserved rather than dropped. Never remove or rename existing fields — downstream extractors and the RAG vector store key off them.

Storage & embedding notes

• One record per JSON file under rag/extracted/. The policy_id is the filename.
• For the vector store, embed two views of each record:
  • The full prose of notable_exclusions_summary + claim_process_summary (high-signal narrative chunks for semantic queries).
  • A flattened key-value string for every populated field (so structured queries like "policies with PED waiting < 24 months" can still match).
• The original policy wordings PDF stays in rag/corpus/ for citation fallback. The schema's source_pdf_path field is the link back.

Chroma chunk metadata

Each chunk persisted in Chroma carries the following metadata keys (set by rag/ingest.py):

Key	Type	Notes
policy_id	str	e.g. aditya-birla__activ-one. Primary filter for per-policy retrieval.
insurer_slug	str	e.g. aditya-birla. Secondary filter.
source_pdf	str	Relative path under rag/corpus/.
page	int	1-indexed PDF page number.
chunk_index	int	Position within the policy's chunk sequence.
doc_type	str	'wordings' / 'brochure' / 'cis' / 'prospectus' / 'curated'. 'curated' (KI-137) marks chunks ingested from hand-curated 40-data/policy_facts/<id>.json rather than raw PDF text.
legacy_issuer	str (optional)	KI-144. Present on indusind-general__* chunks whose source PDFs carry the previous reliance-general issuer branding. Value: 'reliance-general'. Lets retrieval surface legacy citations without breaking the canonical slug.