Upload folder using huggingface_hub

LEARN.md

@@ -0,0 +1,813 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- insurance
+- machine-learning
+- transformers
+- llm
+- qlora
+- ner
+- xgboost
+- glm
+- educational
+- bytical
+---
+
+# Learn INSUREOS from Scratch
+
+This guide explains the INSUREOS project in plain English for users who are new to machine learning, insurance AI, or both. It is based on the walkthrough and explanations we developed while reviewing the real training code in this repository.
+
+## Why this project is split across multiple Hugging Face repos
+
+The INSUREOS project is published in separate repos on Hugging Face because each repo serves a different purpose:
+
+- **Model repos** contain model weights, tokenizer files, metadata, and a model card.
+- **Dataset repo** contains the generated training datasets.
+- **Code repo** contains the actual training, evaluation, serving, and data-generation code.
+- **Search repo** contains the hybrid search engine files and documentation.
+
+This is why a model page may not show Python training scripts directly. The code lives in the shared code repo:
+
+- **Code repo:** [piyushptiwari/insureos-models](https://huggingface.co/piyushptiwari/insureos-models)
+- **Dataset repo:** [piyushptiwari/insureos-training-data](https://huggingface.co/datasets/piyushptiwari/insureos-training-data)
+- **Search repo:** [piyushptiwari/insureos-search-engine](https://huggingface.co/piyushptiwari/insureos-search-engine)
+
+## The 5 main learning modules
+
+We walked through the INSUREOS codebase in the same order someone new to ML would learn it.
+
+### M1. Pricing Models: GLM and EBM
+
+Code:
+- [training/pricing_glm.py](https://huggingface.co/piyushptiwari/insureos-models/blob/main/training/pricing_glm.py)
+- Data: [claims_motor_25000k.csv](https://huggingface.co/datasets/piyushptiwari/insureos-training-data/blob/main/claims_motor_25000k.csv)
+
+What this model does:
+- Predicts insurance premium or claim-related pricing outcomes from tabular policy data.
+
+Core concepts:
+- **Tabular ML**: rows and columns like a spreadsheet.
+- **Feature engineering**: create useful derived variables such as vehicle age and no-claims ratios.
+- **GLM**: a generalized linear model, common in insurance pricing.
+- **Tweedie distribution**: useful when targets behave like a mix of frequency and severity.
+- **EBM**: Explainable Boosting Machine, a glass-box model that stays interpretable.
+
+Why it matters:
+- This is the most traditional insurance ML workflow in the repo.
+- It shows how actuaries and pricing teams often work with structured data.
+
+### M2. Fraud Detection: XGBoost and Isolation Forest
+
+Code:
+- [training/fraud_model.py](https://huggingface.co/piyushptiwari/insureos-models/blob/main/training/fraud_model.py)
+- Data: [claims_all_50000.jsonl](https://huggingface.co/datasets/piyushptiwari/insureos-training-data/blob/main/claims_all_50000.jsonl)
+
+What this model does:
+- Scores claims for fraud risk across Motor, Property, and Liability.
+
+Core concepts:
+- **Decision trees** and **gradient boosting**.
+- **XGBoost** as a strong tabular classifier.
+- **Class imbalance**: fraud is rare, so metrics like accuracy can mislead.
+- **Isolation Forest**: unsupervised anomaly detection for unusual claims.
+- **AUC, precision, recall, F1**: metrics for imbalanced classification.
+
+Important teaching note:
+- In this project, many synthetic datasets score almost perfectly. That is a useful warning sign in ML: perfect performance often means the problem is unrealistically easy, synthetic, or has leakage.
+
+### M3. Document Classification with ModernBERT
+
+Code:
+- [training/doc_classifier.py](https://huggingface.co/piyushptiwari/insureos-models/blob/main/training/doc_classifier.py)
+- Data: [insurance_docs_10k.jsonl](https://huggingface.co/datasets/piyushptiwari/insureos-training-data/blob/main/insurance_docs_10k.jsonl)
+
+What this model does:
+- Reads an insurance document and predicts one of 12 classes, such as policy wording, claim form, or renewal notice.
+
+Core concepts:
+- **Transformers** and **self-attention**.
+- **Tokenization**: breaking text into model-readable pieces.
+- **Embeddings**: turning tokens into vectors.
+- **Sequence classification**: one label for the whole document.
+- **Fine-tuning** a pretrained language model.
+
+Key takeaway:
+- This is the bridge from traditional ML to NLP. The model is no longer using fixed hand-made columns; it learns directly from text.
+
+### M4. Named Entity Recognition with ModernBERT
+
+Code:
+- [training/ner_model.py](https://huggingface.co/piyushptiwari/insureos-models/blob/main/training/ner_model.py)
+- Data: [insurance_ner_8k.jsonl](https://huggingface.co/datasets/piyushptiwari/insureos-training-data/blob/main/insurance_ner_8k.jsonl)
+
+What this model does:
+- Labels each token in insurance text with entity tags such as `PERSON`, `INSURER`, `CLAIM_NUMBER`, or `DATE`.
+
+Core concepts:
+- **Token classification** instead of document classification.
+- **BIO tagging**: `B-`, `I-`, and `O` labels.
+- **Subword alignment**: one word can become multiple tokenizer pieces.
+- **`-100` ignore label** for special tokens and ignored positions.
+- **Seqeval metrics**: entity-level precision, recall, and F1.
+
+Key takeaway:
+- This is how the project turns messy text into structured data fields.
+
+### M5. LLM Fine-Tuning with QLoRA
+
+Code:
+- [training/qlora_finetune.py](https://huggingface.co/piyushptiwari/insureos-models/blob/main/training/qlora_finetune.py)
+- [training/dpo_train.py](https://huggingface.co/piyushptiwari/insureos-models/blob/main/training/dpo_train.py)
+- Data: [insurance_sft_10k.jsonl](https://huggingface.co/datasets/piyushptiwari/insureos-training-data/blob/main/insurance_sft_10k.jsonl)
+- Data: [insurance_dpo_5k.jsonl](https://huggingface.co/datasets/piyushptiwari/insureos-training-data/blob/main/insurance_dpo_5k.jsonl)
+
+What this model does:
+- Fine-tunes Qwen3-4B to generate insurance-domain answers, claims assessments, and other structured text outputs.
+
+Core concepts:
+- **Causal language modeling**: predict the next token.
+- **Autoregressive generation**: write text one token at a time.
+- **Chat templates**: convert system/user/assistant messages into model input.
+- **SFT**: supervised fine-tuning on good examples.
+- **DPO**: preference optimization, teaching the model which answer is better.
+- **QLoRA**: memory-efficient fine-tuning using quantization plus LoRA adapters.
+- **Gradient accumulation** and **gradient checkpointing** to fit large models on smaller GPUs.
+
+Key takeaway:
+- This is the stage where the project becomes a real domain-specific assistant instead of a classifier.
+
+## The mental model for the whole project
+
+Think of INSUREOS as a full insurance AI stack:
+
+- **Data generators** create training datasets.
+- **Training scripts** teach different model types.
+- **Evaluation scripts** measure quality.
+- **Serving code** exposes the models by API.
+- **Search engine** gives retrieval and document search.
+
+Each model handles a different insurance operations task:
+
+- **InsurePricing**: predict price-related outcomes.
+- **InsureFraudNet**: flag suspicious claims.
+- **InsureDocClassifier**: route document types.
+- **InsureNER**: extract structured entities from text.
+- **InsureLLM-4B**: answer and generate insurance-domain text.
+- **InsureSearch**: retrieve the right source material.
+
+## If you are new to ML, learn in this order
+
+1. Start with tabular models: pricing and fraud.
+2. Move to NLP classifiers: document classification.
+3. Then study token labeling: NER.
+4. Then study generation: LLM fine-tuning with QLoRA and DPO.
+5. Finally connect everything with retrieval, serving, and evaluation.
+
+## Where to go next
+
+- Start with the code repo overview: [README.md](https://huggingface.co/piyushptiwari/insureos-models/blob/main/README.md)
+- Explore training scripts: [training/](https://huggingface.co/piyushptiwari/insureos-models/tree/main/training)
+- Explore data generators: [data/](https://huggingface.co/piyushptiwari/insureos-models/tree/main/data)
+- Explore retrieval: [search/](https://huggingface.co/piyushptiwari/insureos-models/tree/main/search)
+
+## Short answer to the original HF question
+
+If you were wondering why model pages did not show all the code and explanations:
+
+- The model repos are intentionally weight-focused.
+- The actual Python code lives in the shared code repo.
+- The learning guide you are reading now is meant to make that split understandable for users.
+# INSUREOS Learning Guide
+
+This guide is for readers who land on the Hugging Face pages and want to understand:
+
+- what each INSUREOS model does
+- which code file trains it
+- what data it uses
+- how the ideas fit together from scratch
+
+It is based on a step-by-step teaching walkthrough of this project, rewritten as a clean reader guide rather than a chat transcript.
+
+## Start Here: Why HF Has Multiple Repos
+
+INSUREOS is intentionally split across separate Hugging Face repos:
+
+- `piyushptiwari/InsureLLM-4B`, `InsureDocClassifier`, `InsureNER`, `InsureFraudNet`, `InsurePricing`: model artifacts and model cards
+- `piyushptiwari/insureos-models`: source code for training, evaluation, serving, and search
+- `piyushptiwari/insureos-training-data`: generated datasets used by the models
+
+That is why a model page may show weights and a README but no Python training files. The code lives in the shared code repo.
+
+## Project Map
+
+If you want the shortest path through the codebase, read these directories in order:
+
+1. [data/](data) for data generation
+2. [training/](training) for model training
+3. [evaluation/](evaluation) for metrics and reports
+4. [search/](search) for the hybrid retrieval engine
+5. [serve/](serve) for inference APIs
+
+## M0. Machine Learning Basics
+
+Before the individual models, here is the minimum mental model:
+
+- A model learns a mapping from inputs to outputs.
+- Training means adjusting parameters to reduce a loss.
+- Evaluation means checking generalization on held-out data.
+- Different tasks need different model families:
+  - tabular regression for pricing
+  - tabular classification for fraud
+  - sequence classification for document type
+  - token classification for NER
+  - causal language modeling for long-form text generation
+
+Three habits matter throughout this repository:
+
+1. Always inspect the data shape before trusting metrics.
+2. Perfect scores on synthetic data are a warning sign, not a victory lap.
+3. Start from the problem type, then choose the model family.
+
+## M1. Pricing Models: GLM and EBM
+
+Training code: [training/pricing_glm.py](training/pricing_glm.py)
+
+Data used:
+
+- [data/output/claims_motor_25000k.csv](data/output/claims_motor_25000k.csv)
+
+Main idea:
+
+- This is a tabular regression problem: predict premium or claim-related value from structured features.
+- The repository trains two pricing models:
+  - Tweedie GLM: the standard actuarial-style linear model for insurance pricing
+  - EBM: an interpretable boosting model with learned shape functions
+
+Core concepts:
+
+- A GLM extends linear regression to non-Gaussian targets.
+- Tweedie is useful in insurance because it can model frequency and severity behavior.
+- A log link means the model predicts on a transformed scale and then maps back.
+- EBM keeps strong performance while staying inspectable.
+
+Features include driver age, years driving, vehicle value, mileage, excesses, claims history, and engineered variables such as:
+
+- `vehicle_age`
+- `driver_experience_ratio`
+- `ncd_ratio`
+
+Why it matters:
+
+- This is the most classical ML in the repo.
+- It teaches feature engineering, target distributions, interpretability, and regression metrics.
+
+## M2. Fraud Detection: XGBoost and Isolation Forest
+
+Training code: [training/fraud_model.py](training/fraud_model.py)
+
+Data used:
+
+- [data/output/claims_all_50000.jsonl](data/output/claims_all_50000.jsonl)
+- [data/output/claims_motor_25000k.csv](data/output/claims_motor_25000k.csv)
+- [data/output/claims_property_15000k.csv](data/output/claims_property_15000k.csv)
+- [data/output/claims_liability_10000k.csv](data/output/claims_liability_10000k.csv)
+
+Main idea:
+
+- This is a tabular classification problem: predict whether a claim is fraudulent.
+- The repository combines two approaches:
+  - XGBoost for supervised fraud probability
+  - Isolation Forest for unsupervised anomaly detection
+
+Core concepts:
+
+- Decision trees split on features to separate normal and suspicious claims.
+- Gradient boosting builds many trees that correct earlier mistakes.
+- Class imbalance matters because fraud is rare.
+- `scale_pos_weight` helps the model take the minority class seriously.
+
+Important practical lesson:
+
+- The saved scores are extremely high on this synthetic data.
+- In real ML work, suspiciously perfect AUC or F1 often means the task is too easy, features leak the answer, or the synthetic data is cleaner than production reality.
+
+Why it matters:
+
+- This module teaches tree-based ML, feature importance, anomaly detection, and rare-event evaluation.
+
+## M3. Document Classification: ModernBERT
+
+Training code: [training/doc_classifier.py](training/doc_classifier.py)
+
+Data used:
+
+- [data/output/insurance_docs_10k.jsonl](data/output/insurance_docs_10k.jsonl)
+
+Main idea:
+
+- This is sequence classification: read a whole document and output one label.
+- The model predicts one of 12 document types such as claim form, policy wording, bordereaux, renewal notice, and FNOL report.
+
+Core concepts:
+
+- A tokenizer converts text into token ids.
+- A transformer encoder produces contextual representations.
+- A classification head maps the final representation to one class.
+- Fine-tuning means starting from a pretrained language model and adapting it to a narrower task.
+
+Why it matters:
+
+- This is the simplest transformer task in the project.
+- It is the bridge from classical ML to deep NLP.
+
+## M4. NER: Label Every Word
+
+Training code: [training/ner_model.py](training/ner_model.py)
+
+Data used:
+
+- [data/output/insurance_ner_8k.jsonl](data/output/insurance_ner_8k.jsonl)
+- [data/constants.py](data/constants.py)
+
+Main idea:
+
+- NER is token classification: instead of assigning one label to the whole text, the model assigns a label to each word or subword.
+- Example entity types include insurer, person, policy number, claim number, regulation, money, date, postcode, and peril.
+
+Core concepts:
+
+- BIO tagging:
+  - `B-XXX` means the beginning of an entity
+  - `I-XXX` means continuation of that entity
+  - `O` means outside any entity
+- Tokenizers split words into subwords, so labels must be aligned carefully.
+- The training code uses `word_ids()` and `-100` to ignore special tokens during loss computation.
+- Entity-level metrics matter more than raw token accuracy.
+
+Why it matters:
+
+- This is how free text becomes structured insurance data.
+- It is a common building block for claims automation and information extraction.
+
+## M5. LLM Fine-Tuning: QLoRA and DPO
+
+Training code:
+
+- [training/qlora_finetune.py](training/qlora_finetune.py)
+- [training/dpo_train.py](training/dpo_train.py)
+- [training/retrain_realworld.py](training/retrain_realworld.py)
+
+Data used:
+
+- [data/output/insurance_sft_10k.jsonl](data/output/insurance_sft_10k.jsonl)
+- [data/output/insurance_dpo_5k.jsonl](data/output/insurance_dpo_5k.jsonl)
+
+Main idea:
+
+- This is causal language modeling: the model writes text one token at a time.
+- Instead of predicting one label, it generates full answers, assessments, and explanations.
+
+The training pipeline has three stages:
+
+1. Supervised fine-tuning (SFT) on insurance conversations
+2. DPO alignment on chosen vs rejected answers
+3. Additional retraining on real-world insurance text converted into instruction data
+
+Core concepts:
+
+- Decoder-only transformers generate the next token autoregressively.
+- QLoRA makes large-model tuning practical on smaller GPUs:
+  - quantize the base model to 4-bit
+  - freeze the base weights
+  - train small LoRA adapters
+- Gradient accumulation simulates a larger batch.
+- Gradient checkpointing reduces memory use.
+- DPO teaches preferences between two answers without a full RLHF pipeline.
+
+Why it matters:
+
+- This is the model family behind long-form assistants.
+- It shows how domain adaptation works when you want the system to speak insurance rather than just classify it.
+
+## Search and Retrieval: InsureSearch
+
+Code:
+
+- [search/api.py](search/api.py)
+- [search/hybrid_engine.py](search/hybrid_engine.py)
+- [search/indexer.py](search/indexer.py)
+- [search/bm25.py](search/bm25.py)
+
+Main idea:
+
+- The LLM should not answer everything from memory.
+- Search retrieves relevant source material first, then the model can answer with better grounding.
+
+INSUREOS search combines:
+
+- dense vector retrieval
+- BM25 keyword retrieval
+- reranking
+- reciprocal rank fusion
+
+Why it matters:
+
+- In production, strong retrieval often matters more than a slightly bigger model.
+
+## How to Read the Repo as a Beginner
+
+If you are new to ML, read in this order:
+
+1. [training/pricing_glm.py](training/pricing_glm.py)
+2. [training/fraud_model.py](training/fraud_model.py)
+3. [training/doc_classifier.py](training/doc_classifier.py)
+4. [training/ner_model.py](training/ner_model.py)
+5. [training/qlora_finetune.py](training/qlora_finetune.py)
+6. [training/dpo_train.py](training/dpo_train.py)
+7. [search/hybrid_engine.py](search/hybrid_engine.py)
+
+That order moves from easier classical ML toward transformer systems and then into retrieval.
+
+## Common Questions
+
+### Why do some metrics look unrealistically perfect?
+
+Because much of the data is synthetic. Synthetic datasets are useful for scaffolding and prototyping, but they are often cleaner and easier than production data.
+
+### Why are there separate model repos and one shared code repo?
+
+Because Hugging Face model pages are best for weights, cards, and usage snippets. A shared code repo avoids duplicating every training script into every model repo.
+
+### Where should I start if I want to run something quickly?
+
+Start with one of these:
+
+- [training/doc_classifier.py](training/doc_classifier.py)
+- [training/ner_model.py](training/ner_model.py)
+- [training/fraud_model.py](training/fraud_model.py)
+
+The LLM path requires more GPU memory and more dependencies.
+
+## Related Repos on Hugging Face
+
+- [InsureLLM-4B](https://huggingface.co/piyushptiwari/InsureLLM-4B)
+- [InsureDocClassifier](https://huggingface.co/piyushptiwari/InsureDocClassifier)
+- [InsureNER](https://huggingface.co/piyushptiwari/InsureNER)
+- [InsureFraudNet](https://huggingface.co/piyushptiwari/InsureFraudNet)
+- [InsurePricing](https://huggingface.co/piyushptiwari/InsurePricing)
+- [insureos-models](https://huggingface.co/piyushptiwari/insureos-models)
+- [insureos-training-data](https://huggingface.co/datasets/piyushptiwari/insureos-training-data)
+
+## Summary
+
+INSUREOS is not one model. It is a compact insurance AI stack:
+
+- tabular ML for pricing and fraud
+- transformer encoders for classification and extraction
+- an LLM for long-form generation
+- a search system for retrieval and grounding
+
+If you are reading this from Hugging Face, the shortest answer is:
+
+- model repos show the artifacts
+- the code repo shows how they were built
+- the dataset repo shows what they were trained on# INSUREOS — Learn How These Models Work
+
+**A plain-English guide to the machine learning behind the INSUREOS model suite.**
+Created by [Bytical AI](https://bytical.ai).
+
+This document is for anyone — engineers, actuaries, claims handlers, students — who
+wants to understand *how* each INSUREOS model actually works, what data trained it,
+and where to find the exact code. No prior ML background assumed: we start from the
+basics and build up.
+
+Every section links to the real training script in this repository so you can read
+the code alongside the explanation.
+
+---
+
+## The big picture: five ML paradigms in one suite
+
+INSUREOS is deliberately built across the whole spectrum of modern ML. Each model is
+a different *family* of algorithm, chosen because it fits the problem:
+
+| # | Model | Problem | ML paradigm | Code | Published model |
+|---|-------|---------|-------------|------|-----------------|
+| 1 | **InsurePricing** | Predict a premium (a number) | Tabular regression (GLM + EBM) | [`training/pricing_glm.py`](training/pricing_glm.py) | [InsurePricing](https://huggingface.co/piyushptiwari/InsurePricing) |
+| 2 | **InsureFraudNet** | Is this claim fraud? (yes/no) | Tabular classification (gradient boosting + anomaly detection) | [`training/fraud_model.py`](training/fraud_model.py) | [InsureFraudNet](https://huggingface.co/piyushptiwari/InsureFraudNet) |
+| 3 | **InsureDocClassifier** | What type of document is this? | Transformer text classification | [`training/doc_classifier.py`](training/doc_classifier.py) | [InsureDocClassifier](https://huggingface.co/piyushptiwari/InsureDocClassifier) |
+| 4 | **InsureNER** | Pull entities out of text | Transformer token classification | [`training/ner_model.py`](training/ner_model.py) | [InsureNER](https://huggingface.co/piyushptiwari/InsureNER) |
+| 5 | **InsureLLM-4B** | Write insurance answers | Large Language Model (QLoRA + DPO) | [`training/qlora_finetune.py`](training/qlora_finetune.py) | [InsureLLM-4B](https://huggingface.co/piyushptiwari/InsureLLM-4B) |
+
+All of them were trained on the open
+[INSUREOS Training Data](https://huggingface.co/datasets/piyushptiwari/insureos-training-data) dataset.
+
+A useful mental model — the difference between the two big groups:
+
+```
+Tabular models (1, 2)         Language models (3, 4, 5)
+-----------------------       -------------------------
+input  = rows of numbers      input  = text
+        (age, value, ...)             ("Aviva policy POL-123 ...")
+engine = decision trees /     engine = transformer (attention)
+         linear equations
+output = a number or a        output = a label, entities,
+         yes/no                        or generated text
+```
+
+---
+
+## Module 1 — InsurePricing: predicting a premium
+
+**The problem.** Given a driver and a vehicle, predict the right premium (a £ amount).
+Predicting a continuous number is called **regression**.
+
+**Concept 1 — Linear regression.** The simplest regression draws a straight line:
+`premium = w1·age + w2·vehicle_value + ... + b`. The model *learns* the weights `w`
+that make the line fit the data best. Easy to interpret, but real insurance costs are
+not a straight line.
+
+**Concept 2 — Generalized Linear Models (GLM).** Insurance has used GLMs for decades
+because they fix two problems with plain linear regression:
+
+- A **link function** lets the relationship curve. We use a **log link**, so the model
+  predicts `log(premium)` and effects become *multiplicative* — "a sports car multiplies
+  the price by 1.4" rather than "adds £400". That matches how risk actually behaves.
+- A **distribution** describes the shape of the target. Claims cost is a spike at £0
+  (most policies never claim) plus a long right tail (a few huge claims). The
+  **Tweedie distribution** models exactly this. We use **power = 1.5**, which sits
+  between Poisson (counting *how often* you claim) and Gamma (*how big* each claim is),
+  so a single model captures *frequency × severity*.
+
+**Concept 3 — Regularization.** The `alpha` parameter discourages the model from
+relying too heavily on any one feature, which prevents **overfitting** (memorising the
+training data instead of learning the pattern).
+
+**Concept 4 — The glass-box second model (EBM).** Alongside the GLM we train an
+**Explainable Boosting Machine** (from Microsoft Research's InterpretML). It is more
+accurate (lower error) yet still fully interpretable: it can show you the exact shape
+of "premium vs driver age". In regulated insurance, being able to *explain* a price is
+as important as the price itself.
+
+**In the code** ([`training/pricing_glm.py`](training/pricing_glm.py)): look for the
+`TweedieRegressor(power=1.5, link='log', alpha=...)` and the `ExplainableBoostingRegressor`.
+Note the engineered features `vehicle_age`, `driver_experience_ratio`, `ncd_ratio` —
+hand-built ratios that give the model more signal than the raw columns alone.
+
+---
+
+## Module 2 — InsureFraudNet: catching fraudulent claims
+
+**The problem.** Given a claim, output a fraud probability. Predicting a yes/no (or a
+probability) is **classification**.
+
+**Concept 1 — Decision trees.** A tree asks a chain of yes/no questions
+("reserve/premium ratio > 5? reported more than 30 days late?") and lands on a verdict.
+Intuitive, but one tree is a weak predictor.
+
+**Concept 2 — Gradient boosting (XGBoost).** Instead of one tree, build *hundreds* of
+small trees **in sequence**, where each new tree focuses on the mistakes the previous
+ones made. The errors get "boosted" away. **XGBoost** is the industry-standard
+implementation and dominates tabular ML competitions. Key settings in our code:
+`max_depth=6`, `learning_rate=0.05`, `n_estimators=500`, `early_stopping=30`
+(stop adding trees once the validation score stops improving).
+
+**Concept 3 — Class imbalance.** Only ~8% of claims are fraud. A lazy model could
+predict "never fraud" and be 92% accurate — and useless. We fix this with
+**`scale_pos_weight ≈ 11.5`** (= number of legit ÷ number of fraud), which tells
+XGBoost to treat each rare fraud case as ~11× more important.
+
+**Concept 4 — Unsupervised anomaly detection (Isolation Forest).** Supervised models
+only catch fraud patterns they have *seen labelled*. An **Isolation Forest** needs no
+labels — it isolates claims that simply *look weird* compared to everything else
+(`contamination=0.08`). This is our safety net for *novel* fraud the labels never
+captured. We train one XGBoost + one Isolation Forest per line of business
+(Motor / Property / Liability).
+
+**Concept 5 — The right metrics.** For rare events, plain accuracy lies. We use:
+
+- **Precision** = of the claims we flagged, how many were really fraud?
+- **Recall** = of the real fraud, how much did we catch?
+- **F1** = the balance of the two.
+- **AUC-PR** (precision-recall area) — far more honest than AUC-ROC when positives are rare.
+
+**In the code** ([`training/fraud_model.py`](training/fraud_model.py)): see
+`XGBClassifier(objective='binary:logistic', eval_metric='aucpr', scale_pos_weight=...)`
+and `IsolationForest(contamination=0.08)`. Engineered features like
+`claim_reserve_ratio`, `claim_premium_ratio`, `new_policy`, `late_report` carry most
+of the predictive power.
+
+---
+
+## Module 3 — InsureDocClassifier: reading whole documents
+
+This is where we leave numbers behind and start working with **text**, using
+**transformers** — the architecture behind all modern language AI.
+
+**Concept 1 — Tokenization.** Computers can't read words, only numbers. A **tokenizer**
+chops text into pieces (tokens — roughly sub-words) and maps each to an integer ID.
+`"Subrogation"` might become `["Sub", "rog", "ation"]` → `[1234, 567, 89]`.
+
+**Concept 2 — Embeddings.** Each token ID becomes a **vector** (a list of numbers) that
+captures meaning. Words used in similar contexts get similar vectors — so the model
+learns that "flood" and "storm" are related.
+
+**Concept 3 — The transformer & self-attention.** The breakthrough idea is
+**self-attention**: when processing a word, the model *looks at every other word in the
+sentence* and decides which ones matter. In "the policy *excludes* flood", attention
+links "excludes" to "flood" so the meaning flips. Stacking many attention layers gives a
+rich, context-aware understanding of the text.
+
+**Concept 4 — The [CLS] token.** A special `[CLS]` token sits at the front of every
+input. After all the attention layers, its vector becomes a **summary of the whole
+document**. We attach a small **classification head** (one linear layer) on top of that
+summary to predict the document type — one of **12 classes** (Policy Schedule, Claim
+Form, FNOL Report, …).
+
+**Concept 5 — Pre-training vs fine-tuning (transfer learning).** We don't train a
+language model from scratch — that costs millions. We start from **ModernBERT**, which
+was already **pre-trained** on huge amounts of text (so it already "knows English"), and
+**fine-tune** it on our 10,000 insurance documents. This **transfer learning** is why a
+specialist model can be built quickly and cheaply.
+
+Key training settings: `MAX_LEN=512`, `EPOCHS=5`, `BATCH=16`, `LR=2e-5`, a cosine
+learning-rate schedule with `warmup_ratio=0.1`, and `weight_decay=0.01`.
+
+- **Epoch** = one full pass over the data. **Batch** = how many examples per step.
+- **Learning rate (LR)** = step size for each update — too big overshoots, too small
+  crawls. **Warmup** ramps the LR up gently at the start. **Weight decay** is
+  regularization that keeps weights small.
+
+**In the code** ([`training/doc_classifier.py`](training/doc_classifier.py)):
+`AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base", num_labels=12)`.
+
+---
+
+## Module 4 — InsureNER: pulling facts out of text
+
+**The shift from Module 3.** Document classification gives **one label for the whole
+text**. NER (Named Entity Recognition) gives **a label for every single word** — this is
+**token classification**.
+
+From a sentence, NER extracts a structured row:
+
+```
+"Brit Insurance must pay Amelia Jones for the vandalism claim (CLM-864344) by 2023-05-25."
+
+INSURER = Brit Insurance    PERSON = Amelia Jones    PERIL = vandalism
+CLAIM_NUMBER = CLM-864344    DATE = 2023-05-25
+```
+
+**Concept 1 — BIO / IOB2 tagging.** Every word gets one of three kinds of tag:
+
+- `O` — **O**utside any entity (a boring word).
+- `B-XXX` — **B**eginning of an entity of type XXX.
+- `I-XXX` — **I**nside (continuation) of that entity.
+
+Example: `Amelia/B-PERSON Jones/I-PERSON`. The `B-`/`I-` split is what lets the model
+tell *two adjacent people* apart from *one two-word person*. Our 13 entity types become
+**27 labels** (`O` + a `B-`/`I-` pair per type).
+
+**Concept 2 — Subword alignment (the tricky part).** The tokenizer may split one word
+into several sub-words (`CLM-864344` → `CL ##M - 864 ##344`), but we only have **one**
+label for the whole word. The function `tokenize_and_align` handles this with three
+rules:
+
+1. Special tokens (`[CLS]`, `[SEP]`, padding) get label **`-100`**, a magic value that
+   tells the loss function "ignore me".
+2. The **first** sub-word of a word gets the real label.
+3. Continuation sub-words convert `B-` → `I-` (the entity is still going).
+
+**Concept 3 — Entity-level metrics (seqeval).** We don't score per token (predicting
+`O` everywhere would score ~85% and be useless). The **seqeval** library scores per
+**entity span** — a prediction only counts if the *whole* entity (type, start, and end)
+is correct. That gives honest precision / recall / F1.
+
+The architecture is the same ModernBERT as Module 3, but the head predicts a label for
+**every** token instead of just the `[CLS]` summary.
+
+**In the code** ([`training/ner_model.py`](training/ner_model.py)):
+`AutoModelForTokenClassification`, the `tokenize_and_align` function, and the
+`seqeval`-based `compute_metrics`. Settings: `MAX_LEN=256`, `EPOCHS=8`, `LR=3e-5`.
+
+---
+
+## Module 5 — InsureLLM-4B: a model that *writes*
+
+The biggest leap. Modules 3–4 **classify** (pick a label). InsureLLM **generates** —
+it writes a full, structured claims assessment paragraph by paragraph.
+
+**Concept 1 — Causal language modelling.** A "causal" LM does one thing: **predict the
+next token** given everything before it. To generate text it predicts a token, appends
+it, feeds the whole thing back, and predicts again — **autoregressive** generation,
+looping hundreds of times. Every ChatGPT-style answer is this loop.
+
+- Modules 3–4 used an **encoder** (BERT) that reads the whole sentence at once.
+- Module 5 uses a **decoder** (Qwen3) that only sees the words to its left.
+
+**Concept 2 — Instruction tuning (SFT).** We show the model thousands of
+`system → user → assistant` chat conversations and train it to produce the *assistant's*
+reply. This is **Supervised Fine-Tuning**. Roles are flattened into one string by the
+tokenizer's **chat template** (`<|im_start|>system … <|im_end|>`), so the model learns
+whose turn it is and when to stop.
+
+**Concept 3 — The size problem.** Qwen3-4B has 4 **billion** parameters. Full
+fine-tuning would need 40+ GB of GPU memory. Two tricks shrink that to fit a single
+16 GB GPU — together they are called **QLoRA**:
+
+- **Quantization (the "Q").** Store each weight in **4 bits** instead of 16
+  (`nf4` format). That is ~4× smaller, so the model fits in ~4–5 GB. The weights are
+  de-quantized to 16-bit only for the actual maths, so accuracy barely drops.
+- **LoRA — Low-Rank Adaptation.** **Freeze all 4 billion original weights** and never
+  touch them. Next to each big matrix, inject two *tiny* trainable matrices `A` and `B`.
+  Instead of learning a giant update, learn the small product `B·A`:
+
+  $$h = Wx + \frac{\alpha}{r}\,(BA)\,x$$
+
+  With `r=64` (the **rank**) and `alpha=128` (a **scaling** factor), only **~1–2% of the
+  parameters are trained**. The huge model keeps all its knowledge (frozen); we just
+  learn a small "insurance accent" on top.
+
+**Concept 4 — Two more memory tricks.**
+- **Gradient accumulation** (`batch=2`, `grad_accum=8` → effective batch 16): add up the
+  gradients from 8 small batches before taking one step, simulating a big batch cheaply.
+- **Gradient checkpointing**: don't store every intermediate value during the forward
+  pass — recompute them during back-prop. Trades compute for memory.
+
+Note `LR=2e-4` here is ~10× higher than the BERT modules: the LoRA adapters start from
+zero and are tiny, so they need a more aggressive learning rate.
+
+**The full InsureLLM pipeline is three stages:** QLoRA SFT → **DPO** alignment
+(teaching the model to prefer the better of two answers — see
+[`training/dpo_train.py`](training/dpo_train.py)) → a final real-world fine-tune.
+
+**In the code** ([`training/qlora_finetune.py`](training/qlora_finetune.py)):
+`BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", ...)`,
+`LoraConfig(r=64, lora_alpha=128, target_modules="all-linear", task_type=CAUSAL_LM)`,
+and the `trl` `SFTTrainer`. At the end, `merge_and_unload()` folds the adapters back
+into the base to produce a single standalone model.
+
+---
+
+## A critical ML lesson: when "perfect" is a red flag
+
+You will notice that several INSUREOS models report **perfect scores** — fraud
+AUC = 1.0, document accuracy = 1.0, NER F1 = 1.0. **In real machine learning, this is a
+warning sign, not a trophy.**
+
+These models are trained on **synthetic** data, where the patterns are cleaner and
+easier than messy reality. A perfect score usually means one of:
+
+- **The task is too easy / the synthetic data is too regular.**
+- **Data leakage** — a feature accidentally gives away the answer.
+
+The instinct to build: **be suspicious of results that look too good.** On real,
+noisy production data, expect roughly **0.80–0.97**, never a flat 1.0. The value of
+this suite is in the *methods and code*, which transfer directly to real data — not in
+the headline 1.0 numbers.
+
+---
+
+## Where to go next
+
+The INSUREOS repository goes beyond these five models:
+
+- **DPO alignment** — [`training/dpo_train.py`](training/dpo_train.py): after SFT teaches
+  the model *what* to say, DPO teaches it *preferences* (choose the better of two answers).
+- **Search / RAG** — the [`search/`](search) directory: a hybrid Vector + BM25 engine
+  that retrieves real documents to ground the LLM's answers (Retrieval-Augmented
+  Generation), reducing hallucination.
+- **Serving & evaluation** — [`serve/api.py`](serve/api.py) and
+  [`evaluation/run_eval.py`](evaluation/run_eval.py): how the models are exposed as an
+  API and measured.
+
+---
+
+## Mini-glossary
+
+| Term | Plain meaning |
+|------|---------------|
+| **Regression** | Predicting a number (e.g. a premium). |
+| **Classification** | Predicting a category (e.g. fraud / not fraud). |
+| **GLM** | Linear model with a link function + a distribution; the actuarial standard. |
+| **Tweedie** | A distribution that models "mostly zero, occasionally huge" — perfect for claims. |
+| **Gradient boosting** | Hundreds of small trees, each fixing the last one's mistakes (XGBoost). |
+| **Class imbalance** | When one outcome (fraud) is rare; needs special handling. |
+| **Tokenization** | Splitting text into integer tokens a model can read. |
+| **Embedding** | A vector of numbers representing a token's meaning. |
+| **Transformer / attention** | Architecture that weighs every word against every other word. |
+| **Fine-tuning** | Adapting a pre-trained model to your specific task. |
+| **Token classification (NER)** | Labelling every word, not the whole text. |
+| **BIO tagging** | `B-`/`I-`/`O` scheme that marks where entities start, continue, and end. |
+| **Causal LM** | A model that generates text by predicting the next token. |
+| **SFT** | Supervised Fine-Tuning on instruction → answer pairs. |
+| **Quantization** | Storing weights in fewer bits to save memory. |
+| **LoRA / QLoRA** | Training tiny adapters on a frozen (quantized) model — cheap fine-tuning. |
+| **Epoch / Batch / LR** | One data pass / examples per step / update step size. |
+
+---
+
+*This guide accompanies the [INSUREOS model suite](https://huggingface.co/piyushptiwari).
+Built by [Bytical AI](https://bytical.ai) — AI agents that run insurance operations.*

README.md

@@ -14,6 +14,12 @@ tags:
 
 **Created by [Bytical AI](https://bytical.ai)** — AI agents that run insurance operations.
 
+## Start Here
+
+If you are trying to understand the Hugging Face repos, start with [LEARN.md](LEARN.md).
+
+That guide explains, in plain English, why the project is split across model, dataset, code, and search repos, and it walks through the concepts, data, and training code behind each model in this repository.
+
 ## Overview
 
 INSUREOS is a complete AI/ML training and inference pipeline for UK insurance operations. This repository contains all source code for data generation, model training, evaluation, data collection, and a hybrid search engine.
@@ -27,7 +33,7 @@ INSUREOS is a complete AI/ML training and inference pipeline for UK insurance op
 | InsureNER | [piyushptiwari/InsureNER](https://huggingface.co/piyushptiwari/InsureNER) | 13-entity NER | F1: 1.0 |
 | InsureFraudNet | [piyushptiwari/InsureFraudNet](https://huggingface.co/piyushptiwari/InsureFraudNet) | Fraud detection (3 LoB) | AUC-ROC: 1.0 |
 | InsurePricing | [piyushptiwari/InsurePricing](https://huggingface.co/piyushptiwari/InsurePricing) | Premium pricing (GLM + EBM) | MAE: £11,132 |
-| InsureSearch | (included in this repo) | Hybrid search engine | 33K docs indexed |
+| InsureSearch | [piyushptiwari/insureos-search-engine](https://huggingface.co/piyushptiwari/insureos-search-engine) | Hybrid search engine | 33K docs indexed |
 
 ### Training Dataset
 

push_to_hf.py

@@ -108,7 +108,7 @@ if __name__ == "__main__":
     # =========================================================
     # 6. Training Code + Search Engine (as a regular repo)
     # =========================================================
-    print("\n[6/7] insureos-models (code repo)")
+    print("\n[6/8] insureos-models (code repo)")
     repo = f"{ORG}/insureos-models"
     ensure_repo(repo, repo_type="model")
     # We upload code only — no model weights, no raw data, no personal files
@@ -128,9 +128,23 @@ if __name__ == "__main__":
     )
 
     # =========================================================
-    # 7. Training Data (as a dataset)
+    # 7. Search Engine (as a regular repo)
     # =========================================================
-    print("\n[7/7] insureos-training-data (dataset)")
+    print("\n[7/8] insureos-search-engine")
+    repo = f"{ORG}/insureos-search-engine"
+    ensure_repo(repo, repo_type="model")
+    upload_folder(
+        repo, f"{BASE}/search", repo_type="model",
+        ignore_patterns=[
+            "__pycache__", "*.pyc", ".DS_Store",
+            "index_data/*",
+        ]
+    )
+
+    # =========================================================
+    # 8. Training Data (as a dataset)
+    # =========================================================
+    print("\n[8/8] insureos-training-data (dataset)")
     repo = f"{ORG}/insureos-training-data"
     ensure_repo(repo, repo_type="dataset")
     upload_folder(
@@ -149,5 +163,7 @@ if __name__ == "__main__":
     print(f"  https://huggingface.co/{ORG}/InsurePricing")
     print(f"\nCode:")
     print(f"  https://huggingface.co/{ORG}/insureos-models")
+    print(f"\nSearch:")
+    print(f"  https://huggingface.co/{ORG}/insureos-search-engine")
     print(f"\nDataset:")
     print(f"  https://huggingface.co/datasets/{ORG}/insureos-training-data")

search/README.md

@@ -0,0 +1,54 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- insurance
+- search
+- rag
+- bm25
+- qdrant
+- fastapi
+- retrieval
+- bytical
+pipeline_tag: sentence-similarity
+---
+
+# InsureSearch — Hybrid Insurance Search Engine
+
+InsureSearch is the retrieval layer of INSUREOS. It combines vector search, BM25 keyword search, reranking, and FastAPI serving for insurance knowledge retrieval and RAG pipelines.
+
+## What it does
+
+- Indexes insurance documents and chunks.
+- Supports semantic, keyword, and hybrid retrieval.
+- Reranks results for better relevance.
+- Powers retrieval for insurance assistants and internal search workflows.
+
+## Main Components
+
+- `embedder.py` — embeddings for semantic search.
+- `bm25.py` — keyword retrieval.
+- `vector_store.py` — vector storage and retrieval.
+- `reranker.py` — reranking stage.
+- `hybrid_engine.py` — fuses vector and keyword results.
+- `indexer.py` — builds the search index.
+- `api.py` — FastAPI endpoints.
+
+## Code and Learning Guide
+
+- **Search code:** [piyushptiwari/insureos-models/tree/main/search](https://huggingface.co/piyushptiwari/insureos-models/tree/main/search)
+- **Full code repo:** [piyushptiwari/insureos-models](https://huggingface.co/piyushptiwari/insureos-models)
+- **Learning guide:** [LEARN.md](https://huggingface.co/piyushptiwari/insureos-models/blob/main/LEARN.md)
+
+## Relationship to the models
+
+InsureSearch is typically used together with:
+
+- [InsureLLM-4B](https://huggingface.co/piyushptiwari/InsureLLM-4B) for retrieval-augmented generation.
+- [InsureDocClassifier](https://huggingface.co/piyushptiwari/InsureDocClassifier) for document routing.
+- [InsureNER](https://huggingface.co/piyushptiwari/InsureNER) for structured extraction before indexing.
+
+## Why this repo exists separately on Hugging Face
+
+Users often expect a model card link when a model mentions a companion search engine. Publishing this folder as its own HF repo makes that link explicit and avoids broken references.