feat: Add comprehensive data pipeline and fine-tuning

Data Pipeline (scripts/data_pipeline/):
- step1_unify.py: Unifies MBOX, JSON, CSV, XML sources
- step2_filter.py: Removes OTPs, spam, marketing
- step3_baseline.py: Tests regex extractor accuracy
- step4_label.py: Creates labeled training data

Synthetic Data Generator:
- generate_synthetic.py: Production-grade grammar-based generator
- 100K+ realistic Indian bank transactions
- All major banks (HDFC, ICICI, SBI, Axis, Kotak, etc.)
- Brokerages (Zerodha, Groww, Upstox, etc.)
- E-commerce, food, travel, utilities, entertainment
- generate_advanced.py: Advanced features
- Markov Chain for realistic message flow
- Real data calibration
- Multilingual (Hindi, Tamil, Telugu, Bengali, Kannada)
- Data augmentation and edge case oversampling

Fine-tuning Pipeline (scripts/finetune.py):
- Supports MLX (Apple Silicon) and PyTorch
- LoRA fine-tuning with automatic data prep
- Model fusion and evaluation

Training Results:
- 152K training records
- Val loss: 2.42 -> 0.46 (81% reduction)
- 100% JSON parsing on test cases
- Multilingual support working

scripts/data_pipeline/README.md

@@ -0,0 +1,128 @@
+# Data Pipeline
+
+3-step pipeline to prepare training data for FinEE model fine-tuning.
+
+## Overview
+
+```
+Raw Data (XML/JSON/CSV/MBOX)
+         │
+         ▼
+┌────────────────────────┐
+│ Step 1: Unify          │  → step1_unified.csv
+│ Standardize formats    │     (all records, single schema)
+└────────────────────────┘
+         │
+         ▼
+┌────────────────────────┐
+│ Step 2: Filter         │  → step2_training_ready.csv
+│ Remove garbage         │     (clean transactions only)
+└────────────────────────┘
+         │
+         ▼
+┌────────────────────────┐
+│ Step 3: Baseline       │  → step3_baseline_results.csv
+│ Test current accuracy  │     (extracted fields + metrics)
+└────────────────────────┘
+```
+
+## Quick Start
+
+### 1. Copy your data to the workspace
+
+```bash
+# Example: Google Takeout export
+cp -r ~/Downloads/Takeout /Users/ranjit/llm-mail-trainer/data/raw/
+
+# Or SMS backup
+cp sms_backup.xml /Users/ranjit/llm-mail-trainer/data/raw/
+```
+
+### 2. Run the pipeline
+
+```bash
+cd /Users/ranjit/llm-mail-trainer
+
+# Step 1: Unify all data formats
+python scripts/data_pipeline/step1_unify.py --input data/raw/ --output data/pipeline/step1_unified.csv
+
+# Step 2: Filter garbage (OTPs, spam, etc.)
+python scripts/data_pipeline/step2_filter.py --input data/pipeline/step1_unified.csv
+
+# Step 3: Test baseline accuracy
+python scripts/data_pipeline/step3_baseline.py
+```
+
+## Supported Input Formats
+
+| Format | Source | Example |
+|--------|--------|---------|
+| `.mbox` | Gmail export | Mail.mbox |
+| `.json` | Google Takeout | transactions.json |
+| `.csv` | Bank exports | statements.csv |
+| `.xml` | SMS Backup apps | sms_backup.xml |
+
+## Output Schema
+
+All data is standardized to:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `timestamp` | string | When message was received |
+| `sender` | string | Bank/sender name |
+| `body` | string | Message content |
+| `source` | string | Original file source |
+
+## Step 2 Filters
+
+### Messages REMOVED (Garbage):
+- OTPs / Verification codes
+- Login alerts
+- Marketing spam (% off, offers)
+- Bill reminders (not transactions)
+- Account statements
+- Delivery notifications
+
+### Messages KEPT (Transactions):
+- Debit/Credit notifications
+- UPI payments
+- NEFT/IMPS transfers
+- Amount + account references
+
+## Step 3 Metrics
+
+The baseline test measures:
+
+| Metric | Description |
+|--------|-------------|
+| Extraction Success Rate | % of messages where amount + type extracted |
+| Field Coverage | % for each field (amount, merchant, etc.) |
+| Confidence Distribution | LOW / MEDIUM / HIGH breakdown |
+| Top Merchants | Most common extracted merchants |
+| Processing Speed | Messages per second |
+
+## Directory Structure
+
+```
+data/
+├── raw/                    # Put your raw data here
+│   └── Takeout/
+│       └── ...
+├── pipeline/               # Pipeline outputs
+│   ├── step1_unified.csv
+│   ├── step2_training_ready.csv
+│   ├── step2_garbage.csv       (optional)
+│   ├── step2_uncertain.csv     (optional)
+│   ├── step3_baseline_results.csv
+│   └── step3_baseline_analysis.json
+└── training/               # Final training data (after labeling)
+```
+
+## Next Steps
+
+After Step 3:
+
+1. Review low-confidence extractions
+2. Add ground truth labels for training
+3. Identify patterns that need new regex
+4. Fine-tune the LLM on labeled data

scripts/data_pipeline/generate_advanced.py

@@ -0,0 +1,1149 @@
+#!/usr/bin/env python3
+"""
+Advanced Synthetic Data Generator v4.0
+======================================
+
+New Features:
+1. Markov Chain for realistic message flow
+2. Real data calibration from actual samples
+3. Multilingual support (Hindi, Tamil, Telugu, Bengali, Kannada)
+4. PDF/Image generation for document training
+5. Statistical augmentation for rare edge cases
+
+Author: Ranjit Behera
+"""
+
+from __future__ import annotations
+import json
+import random
+import hashlib
+import argparse
+import math
+import pickle
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field, asdict
+from datetime import datetime, timedelta, date
+from decimal import Decimal, ROUND_HALF_UP
+from enum import Enum, auto
+from pathlib import Path
+from typing import List, Dict, Optional, Tuple, Set, Any, Iterator
+from collections import defaultdict, Counter
+import re
+
+
+# ============================================================================
+# MARKOV CHAIN MESSAGE GENERATOR
+# ============================================================================
+
+class MarkovChain:
+    """
+    Markov Chain for realistic message structure.
+    
+    Learns transition probabilities from real data to generate
+    messages that follow actual patterns.
+    """
+    
+    def __init__(self, order: int = 2):
+        """
+        Args:
+            order: n-gram order (1 = unigram, 2 = bigram, etc.)
+        """
+        self.order = order
+        self.transitions: Dict[Tuple, Counter] = defaultdict(Counter)
+        self.start_states: Counter = Counter()
+        
+    def train(self, messages: List[str]):
+        """Train on real messages."""
+        for message in messages:
+            tokens = self._tokenize(message)
+            if len(tokens) <= self.order:
+                continue
+            
+            # Start state
+            start = tuple(tokens[:self.order])
+            self.start_states[start] += 1
+            
+            # Build transitions
+            for i in range(len(tokens) - self.order):
+                state = tuple(tokens[i:i + self.order])
+                next_token = tokens[i + self.order]
+                self.transitions[state][next_token] += 1
+    
+    def _tokenize(self, text: str) -> List[str]:
+        """Tokenize preserving structure."""
+        # Split on whitespace but keep special tokens
+        tokens = []
+        for word in text.split():
+            # Keep amounts as single tokens
+            if re.match(r'^Rs\.?\d', word) or re.match(r'^₹\d', word):
+                tokens.append(word)
+            # Keep VPAs together
+            elif '@' in word:
+                tokens.append(word)
+            else:
+                tokens.append(word)
+        return tokens
+    
+    def generate(self, rng: random.Random, max_length: int = 50) -> str:
+        """Generate a message using learned transitions."""
+        if not self.start_states:
+            return ""
+        
+        # Sample start state
+        states = list(self.start_states.keys())
+        weights = [self.start_states[s] for s in states]
+        current = list(rng.choices(states, weights=weights)[0])
+        
+        result = list(current)
+        
+        for _ in range(max_length - len(current)):
+            state = tuple(current[-self.order:])
+            
+            if state not in self.transitions:
+                break
+            
+            # Sample next token
+            next_tokens = list(self.transitions[state].keys())
+            weights = [self.transitions[state][t] for t in next_tokens]
+            next_token = rng.choices(next_tokens, weights=weights)[0]
+            
+            result.append(next_token)
+            current.append(next_token)
+        
+        return ' '.join(result)
+    
+    def save(self, path: Path):
+        """Save trained model."""
+        with open(path, 'wb') as f:
+            pickle.dump({
+                'order': self.order,
+                'transitions': dict(self.transitions),
+                'start_states': dict(self.start_states),
+            }, f)
+    
+    @classmethod
+    def load(cls, path: Path) -> 'MarkovChain':
+        """Load trained model."""
+        with open(path, 'rb') as f:
+            data = pickle.load(f)
+        
+        chain = cls(order=data['order'])
+        chain.transitions = defaultdict(Counter, {
+            k: Counter(v) for k, v in data['transitions'].items()
+        })
+        chain.start_states = Counter(data['start_states'])
+        return chain
+
+
+class HybridGenerator:
+    """
+    Combines Markov Chain with template-based generation.
+    
+    Uses Markov for structure, templates for entity placement.
+    """
+    
+    def __init__(self, markov: MarkovChain):
+        self.markov = markov
+        self.entity_patterns = {
+            'AMOUNT': r'Rs\.?\s*[\d,]+(?:\.\d{2})?',
+            'ACCOUNT': r'XX\d{4}',
+            'DATE': r'\d{1,2}[-/]\d{1,2}[-/]\d{2,4}',
+            'REF': r'\d{12,16}',
+            'VPA': r'[a-z0-9]+@[a-z]+',
+        }
+    
+    def generate(self, entities: Dict[str, str], rng: random.Random) -> str:
+        """
+        Generate message with specific entities.
+        
+        1. Generate base structure from Markov
+        2. Replace placeholders with actual entities
+        """
+        base = self.markov.generate(rng)
+        
+        # Replace detected patterns with actual entities
+        for entity_type, pattern in self.entity_patterns.items():
+            if entity_type in entities:
+                base = re.sub(pattern, entities[entity_type], base, count=1)
+        
+        return base
+
+
+# ============================================================================
+# REAL DATA CALIBRATION
+# ============================================================================
+
+@dataclass
+class DistributionFit:
+    """Fitted statistical distribution."""
+    name: str
+    params: Dict[str, float]
+    
+    def sample(self, rng: random.Random) -> float:
+        """Sample from fitted distribution."""
+        if self.name == 'normal':
+            value = rng.gauss(self.params['mean'], self.params['std'])
+            return max(self.params.get('min', 0), 
+                      min(self.params.get('max', float('inf')), value))
+        
+        elif self.name == 'lognormal':
+            # Log-normal is better for amounts
+            log_value = rng.gauss(self.params['mu'], self.params['sigma'])
+            return math.exp(log_value)
+        
+        elif self.name == 'exponential':
+            return rng.expovariate(1 / self.params['lambda'])
+        
+        elif self.name == 'uniform':
+            return rng.uniform(self.params['min'], self.params['max'])
+        
+        elif self.name == 'categorical':
+            items = list(self.params['categories'].keys())
+            weights = list(self.params['categories'].values())
+            return rng.choices(items, weights=weights)[0]
+        
+        return 0
+
+
+class DataCalibrator:
+    """
+    Calibrate synthetic distributions to match real data.
+    
+    Fits statistical distributions to actual transaction data.
+    """
+    
+    def __init__(self):
+        self.amount_dist: Optional[DistributionFit] = None
+        self.category_dist: Optional[DistributionFit] = None
+        self.bank_dist: Optional[DistributionFit] = None
+        self.hour_dist: Optional[DistributionFit] = None
+        
+        # Amount by category
+        self.amount_by_category: Dict[str, DistributionFit] = {}
+    
+    def fit_from_data(self, data: List[Dict]):
+        """Fit distributions from real data."""
+        if not data:
+            return
+        
+        # Extract amounts
+        amounts = [r['amount'] for r in data if r.get('amount')]
+        if amounts:
+            mean_amt = sum(amounts) / len(amounts)
+            var_amt = sum((x - mean_amt) ** 2 for x in amounts) / len(amounts)
+            std_amt = math.sqrt(var_amt)
+            
+            # Use log-normal for amounts (always positive, right-skewed)
+            log_amounts = [math.log(max(1, a)) for a in amounts]
+            mu = sum(log_amounts) / len(log_amounts)
+            sigma_sq = sum((x - mu) ** 2 for x in log_amounts) / len(log_amounts)
+            
+            self.amount_dist = DistributionFit(
+                name='lognormal',
+                params={'mu': mu, 'sigma': math.sqrt(sigma_sq)}
+            )
+        
+        # Fit categories
+        categories = Counter(r.get('category') for r in data if r.get('category'))
+        if categories:
+            total = sum(categories.values())
+            self.category_dist = DistributionFit(
+                name='categorical',
+                params={'categories': {k: v/total for k, v in categories.items()}}
+            )
+        
+        # Fit banks
+        banks = Counter(r.get('bank') for r in data if r.get('bank'))
+        if banks:
+            total = sum(banks.values())
+            self.bank_dist = DistributionFit(
+                name='categorical',
+                params={'categories': {k: v/total for k, v in banks.items()}}
+            )
+        
+        # Fit amounts by category
+        by_category = defaultdict(list)
+        for r in data:
+            if r.get('amount') and r.get('category'):
+                by_category[r['category']].append(r['amount'])
+        
+        for cat, amounts in by_category.items():
+            if len(amounts) >= 10:
+                log_amounts = [math.log(max(1, a)) for a in amounts]
+                mu = sum(log_amounts) / len(log_amounts)
+                sigma_sq = sum((x - mu) ** 2 for x in log_amounts) / len(log_amounts)
+                
+                self.amount_by_category[cat] = DistributionFit(
+                    name='lognormal',
+                    params={'mu': mu, 'sigma': max(0.1, math.sqrt(sigma_sq))}
+                )
+    
+    def sample_amount(self, category: Optional[str], rng: random.Random) -> float:
+        """Sample amount, optionally by category."""
+        if category and category in self.amount_by_category:
+            return self.amount_by_category[category].sample(rng)
+        elif self.amount_dist:
+            return self.amount_dist.sample(rng)
+        else:
+            return rng.uniform(100, 10000)
+    
+    def sample_category(self, rng: random.Random) -> str:
+        """Sample category from fitted distribution."""
+        if self.category_dist:
+            return self.category_dist.sample(rng)
+        return 'shopping'
+    
+    def sample_bank(self, rng: random.Random) -> str:
+        """Sample bank from fitted distribution."""
+        if self.bank_dist:
+            return self.bank_dist.sample(rng)
+        return 'HDFC'
+    
+    def save(self, path: Path):
+        """Save calibration."""
+        with open(path, 'wb') as f:
+            pickle.dump({
+                'amount_dist': self.amount_dist,
+                'category_dist': self.category_dist,
+                'bank_dist': self.bank_dist,
+                'amount_by_category': self.amount_by_category,
+            }, f)
+    
+    @classmethod
+    def load(cls, path: Path) -> 'DataCalibrator':
+        """Load calibration."""
+        with open(path, 'rb') as f:
+            data = pickle.load(f)
+        
+        calibrator = cls()
+        calibrator.amount_dist = data.get('amount_dist')
+        calibrator.category_dist = data.get('category_dist')
+        calibrator.bank_dist = data.get('bank_dist')
+        calibrator.amount_by_category = data.get('amount_by_category', {})
+        return calibrator
+
+
+# ============================================================================
+# MULTILINGUAL SUPPORT
+# ============================================================================
+
+class Language(Enum):
+    ENGLISH = "en"
+    HINDI = "hi"
+    TAMIL = "ta"
+    TELUGU = "te"
+    BENGALI = "bn"
+    KANNADA = "kn"
+    MARATHI = "mr"
+    GUJARATI = "gu"
+
+
+@dataclass
+class MultilingualTemplate:
+    """Template with translations."""
+    english: str
+    translations: Dict[Language, str]
+    
+    def get(self, lang: Language) -> str:
+        """Get template in specified language."""
+        if lang == Language.ENGLISH:
+            return self.english
+        return self.translations.get(lang, self.english)
+
+
+class MultilingualBank:
+    """
+    Bank SMS templates in multiple Indian languages.
+    
+    Based on actual bank SMS formats in different languages.
+    """
+    
+    TEMPLATES = {
+        'debit': MultilingualTemplate(
+            english="{bank}: Rs.{amount} debited from A/c XX{account} on {date}. {vpa}. Ref: {ref}",
+            translations={
+                Language.HINDI: "{bank}: आपके खाते XX{account} से Rs.{amount} डेबिट हुआ। दिनांक {date}। {vpa}। संदर्भ: {ref}",
+                Language.TAMIL: "{bank}: உங்கள் கணக்கு XX{account} இல் இருந்து Rs.{amount} டெபிட் செய்யப்பட்டது. தேதி {date}. Ref: {ref}",
+                Language.TELUGU: "{bank}: మీ ఖాతా XX{account} నుండి Rs.{amount} డెబిట్ చేయబడింది. తేదీ {date}. {vpa}. Ref: {ref}",
+                Language.BENGALI: "{bank}: আপনার অ্যাকাউন্ট XX{account} থেকে Rs.{amount} ডেবিট হয়েছে। তারিখ {date}। Ref: {ref}",
+                Language.KANNADA: "{bank}: ನಿಮ್ಮ ಖಾತೆ XX{account} ನಿಂದ Rs.{amount} ಡೆಬಿಟ್ ಆಗಿದೆ. ದಿನಾಂಕ {date}. Ref: {ref}",
+                Language.MARATHI: "{bank}: तुमच्या खात्यातून XX{account} Rs.{amount} डेबिट झाले. तारीख {date}. Ref: {ref}",
+                Language.GUJARATI: "{bank}: તમારા ખાતા XX{account} માંથી Rs.{amount} ડેબિટ થયું. તારીખ {date}. Ref: {ref}",
+            }
+        ),
+        'credit': MultilingualTemplate(
+            english="{bank}: Rs.{amount} credited to A/c XX{account} on {date}. {sender}. Ref: {ref}",
+            translations={
+                Language.HINDI: "{bank}: आपके खाते XX{account} में Rs.{amount} क्रेडिट हुआ। दिनांक {date}। {sender}। संदर्भ: {ref}",
+                Language.TAMIL: "{bank}: உங்கள் கணக்கு XX{account} க்கு Rs.{amount} கிரெடிட் செய்யப்பட்டது. தேதி {date}. Ref: {ref}",
+                Language.TELUGU: "{bank}: మీ ఖాతా XX{account} కు Rs.{amount} క్రెడిట్ చేయబడింది. తేదీ {date}. Ref: {ref}",
+                Language.BENGALI: "{bank}: আপনার অ্যাকাউন্ট XX{account} এ Rs.{amount} ক্রেডিট হয়েছে। তারিখ {date}। Ref: {ref}",
+                Language.KANNADA: "{bank}: ನಿಮ್ಮ ಖಾತೆ XX{account} ಗೆ Rs.{amount} ಕ್ರೆಡಿಟ್ ಆಗಿದೆ. ದಿನಾಂಕ {date}. Ref: {ref}",
+                Language.MARATHI: "{bank}: तुमच्या खात्यात XX{account} Rs.{amount} क्रेडिट झाले. तारीख {date}. Ref: {ref}",
+                Language.GUJARATI: "{bank}: તમારા ખાતા XX{account} માં Rs.{amount} ક્રેડિટ થયું. તારીખ {date}. Ref: {ref}",
+            }
+        ),
+        'otp': MultilingualTemplate(
+            english="{bank}: Your OTP is {otp}. Valid for 10 mins. Do not share with anyone.",
+            translations={
+                Language.HINDI: "{bank}: आपका OTP {otp} है। 10 मिनट के लिए मान्य। किसी के साथ साझा न करें।",
+                Language.TAMIL: "{bank}: உங்கள் OTP {otp}. 10 நிமிடங்களுக்கு செல்லுபடியாகும். யாருடனும் பகிர வேண்டாம்.",
+                Language.TELUGU: "{bank}: మీ OTP {otp}. 10 నిమిషాలు చెల్లుబాటు. ఎవరితోనూ షేర్ చేయకండి.",
+                Language.BENGALI: "{bank}: আপনার OTP হল {otp}। 10 মিনিটের জন্য বৈধ। কারো সাথে শেয়ার করবেন না।",
+            }
+        ),
+        'balance': MultilingualTemplate(
+            english="{bank}: Your A/c XX{account} balance is Rs.{balance}.",
+            translations={
+                Language.HINDI: "{bank}: आपके खाते XX{account} में शेष राशि Rs.{balance} है।",
+                Language.TAMIL: "{bank}: உங்கள் கணக்கு XX{account} இருப்பு Rs.{balance}.",
+                Language.TELUGU: "{bank}: మీ ఖాతా XX{account} బ్యాలెన్స్ Rs.{balance}.",
+                Language.BENGALI: "{bank}: আপনার অ্যাকাউন্ট XX{account} ব্যালেন্স Rs.{balance}।",
+            }
+        ),
+    }
+    
+    # Numbers in Indian languages
+    NUMBERS = {
+        Language.HINDI: {
+            '0': '०', '1': '१', '2': '२', '3': '३', '4': '४',
+            '5': '५', '6': '६', '7': '७', '8': '८', '9': '९',
+        },
+        Language.BENGALI: {
+            '0': '০', '1': '১', '2': '২', '3': '৩', '4': '৪',
+            '5': '৫', '6': '৬', '7': '৭', '8': '৮', '9': '৯',
+        },
+        Language.TAMIL: {
+            '0': '௦', '1': '௧', '2': '௨', '3': '௩', '4': '௪',
+            '5': '௫', '6': '௬', '7': '௭', '8': '௮', '9': '௯',
+        },
+        Language.KANNADA: {
+            '0': '೦', '1': '೧', '2': '೨', '3': '೩', '4': '೪',
+            '5': '೫', '6': '೬', '7': '೭', '8': '೮', '9': '೯',
+        },
+    }
+    
+    @classmethod
+    def generate(
+        cls,
+        template_type: str,
+        language: Language,
+        params: Dict[str, str],
+        use_native_numbers: bool = False,
+        rng: random.Random = None
+    ) -> str:
+        """Generate message in specified language."""
+        template = cls.TEMPLATES.get(template_type)
+        if not template:
+            return ""
+        
+        text = template.get(language)
+        message = text.format(**params)
+        
+        # Optionally convert numbers to native script
+        if use_native_numbers and language in cls.NUMBERS:
+            for eng, native in cls.NUMBERS[language].items():
+                message = message.replace(eng, native)
+        
+        return message
+
+
+class MultilingualNameGenerator:
+    """Generate names in multiple Indian languages."""
+    
+    NAMES = {
+        Language.HINDI: [
+            "राहुल शर्मा", "प्रिया सिंह", "अमित कुमार", "नेहा गुप्ता",
+            "विजय पटेल", "दीपक वर्मा", "अंजलि मेहता", "राजेश नायर",
+            "सुनीता अय्यर", "अरुण जोशी", "पूजा रेड्डी", "संजय मिश्रा",
+        ],
+        Language.TAMIL: [
+            "முருகன் செல்வம்", "லக்ஷ்மி நாராயணன்", "கார்த்திக் சுப்பிரமணியம்",
+            "மீனா குமார்", "அருண் பிரகாஷ்", "சரிதா வேணுகோபால்",
+        ],
+        Language.TELUGU: [
+            "రవి కుమార్", "లక్ష్మీ దేవి", "సురేష్ రెడ్డి", "వెంకట రావు",
+            "ప్రసాద్ నాయుడు", "కమల శర్మ", "రాజేష్ గుప్తా",
+        ],
+        Language.BENGALI: [
+            "রাহুল ব্যানার্জী", "���্রিয়া দাস", "অমিত চক্রবর্তী",
+            "সুমিতা সেন", "রাজেশ মুখার্জী", "কবিতা বসু",
+        ],
+        Language.KANNADA: [
+            "ರಾಜೇಶ್ ಗೌಡ", "ಲಕ್ಷ್ಮೀ ನಾರಾಯಣ", "ಸುರೇಶ್ ಕುಮಾರ್",
+            "ಮೀನಾ ಹೆಗ್ಡೆ", "ಪ್ರಕಾಶ್ ರಾವ್", "ನೇತ್ರಾ ಶೆಟ್ಟಿ",
+        ],
+    }
+    
+    @classmethod
+    def get_name(cls, language: Language, rng: random.Random) -> str:
+        """Get a random name in specified language."""
+        names = cls.NAMES.get(language, cls.NAMES[Language.HINDI])
+        return rng.choice(names)
+
+
+# ============================================================================
+# DATA AUGMENTATION
+# ============================================================================
+
+class DataAugmenter:
+    """
+    Advanced data augmentation techniques.
+    
+    Techniques:
+    1. Back-translation (via templates)
+    2. Synonym replacement
+    3. Random insertion/deletion
+    4. Noise injection
+    5. Entity swapping
+    """
+    
+    SYNONYMS = {
+        'debited': ['withdrawn', 'deducted', 'paid', 'transferred', 'sent'],
+        'credited': ['received', 'deposited', 'added', 'transferred'],
+        'transaction': ['payment', 'transfer', 'txn'],
+        'account': ['A/c', 'Acc', 'Acct', 'a/c'],
+        'reference': ['Ref', 'UTR', 'Txn ID'],
+        'available': ['Avl', 'remaining', 'left'],
+        'balance': ['Bal', 'amt'],
+    }
+    
+    def __init__(self, seed: int = 42):
+        self.rng = random.Random(seed)
+    
+    def augment(
+        self,
+        text: str,
+        ground_truth: Dict,
+        techniques: List[str] = None
+    ) -> List[Tuple[str, Dict]]:
+        """
+        Generate augmented versions of a sample.
+        
+        Returns list of (augmented_text, ground_truth) tuples.
+        """
+        if techniques is None:
+            techniques = ['synonym', 'noise', 'case']
+        
+        augmented = []
+        
+        if 'synonym' in techniques:
+            aug = self._synonym_replace(text)
+            augmented.append((aug, ground_truth))
+        
+        if 'noise' in techniques:
+            aug = self._add_noise(text)
+            augmented.append((aug, ground_truth))
+        
+        if 'case' in techniques:
+            aug = self._vary_case(text)
+            augmented.append((aug, ground_truth))
+        
+        if 'truncate' in techniques:
+            aug = self._truncate(text)
+            augmented.append((aug, ground_truth))
+        
+        if 'reorder' in techniques:
+            aug = self._reorder_phrases(text)
+            augmented.append((aug, ground_truth))
+        
+        return augmented
+    
+    def _synonym_replace(self, text: str) -> str:
+        """Replace words with synonyms."""
+        words = text.split()
+        for i, word in enumerate(words):
+            word_lower = word.lower().strip('.,;:')
+            if word_lower in self.SYNONYMS and self.rng.random() < 0.3:
+                synonym = self.rng.choice(self.SYNONYMS[word_lower])
+                # Preserve case
+                if word[0].isupper():
+                    synonym = synonym.capitalize()
+                words[i] = synonym
+        return ' '.join(words)
+    
+    def _add_noise(self, text: str) -> str:
+        """Add realistic noise."""
+        # Random spacing
+        if self.rng.random() < 0.3:
+            text = text.replace('. ', '.')
+        if self.rng.random() < 0.3:
+            text = text.replace(': ', ':')
+        
+        # Abbreviations
+        text = text.replace('Reference', 'Ref' if self.rng.random() < 0.5 else 'Reference')
+        text = text.replace('Account', 'A/c' if self.rng.random() < 0.5 else 'Account')
+        
+        return text
+    
+    def _vary_case(self, text: str) -> str:
+        """Vary text case."""
+        r = self.rng.random()
+        if r < 0.2:
+            return text.upper()
+        elif r < 0.4:
+            return text.lower()
+        return text
+    
+    def _truncate(self, text: str) -> str:
+        """Truncate to SMS limit."""
+        if len(text) > 160:
+            return text[:157] + '...'
+        return text
+    
+    def _reorder_phrases(self, text: str) -> str:
+        """Reorder independent phrases."""
+        # Split by common delimiters
+        phrases = re.split(r'[.;]', text)
+        phrases = [p.strip() for p in phrases if p.strip()]
+        
+        if len(phrases) <= 2:
+            return text
+        
+        # Keep first phrase, shuffle middle, keep last
+        first = phrases[0]
+        last = phrases[-1]
+        middle = phrases[1:-1]
+        self.rng.shuffle(middle)
+        
+        return '. '.join([first] + middle + [last])
+    
+    def augment_batch(
+        self,
+        data: List[Dict],
+        augmentation_factor: int = 3
+    ) -> List[Dict]:
+        """Augment entire dataset."""
+        augmented_data = []
+        
+        for record in data:
+            text = record.get('text') or record.get('input', '')
+            gt = record.get('ground_truth', record.get('output', {}))
+            
+            if isinstance(gt, str):
+                gt = json.loads(gt)
+            
+            # Original
+            augmented_data.append(record)
+            
+            # Augmented versions
+            for aug_text, aug_gt in self.augment(text, gt)[:augmentation_factor-1]:
+                augmented_data.append({
+                    'text': aug_text,
+                    'ground_truth': aug_gt,
+                    'augmented': True,
+                })
+        
+        return augmented_data
+
+
+# ============================================================================
+# RARE EDGE CASE OVERSAMPLING
+# ============================================================================
+
+class RareEdgeCaseSampler:
+    """
+    Oversample rare edge cases to improve model robustness.
+    
+    Uses importance sampling to increase representation of:
+    - Failed transactions
+    - Large amounts
+    - Unusual formats
+    - Rare banks
+    - Unicode text
+    """
+    
+    def __init__(self, seed: int = 42):
+        self.rng = random.Random(seed)
+        
+        # Define edge case conditions
+        self.edge_cases = {
+            'failed_txn': lambda r: r.get('status') == 'failed',
+            'pending_txn': lambda r: r.get('status') == 'pending',
+            'large_amount': lambda r: (r.get('amount') or 0) > 100000,
+            'small_amount': lambda r: (r.get('amount') or float('inf')) < 10,
+            'unicode': lambda r: any(ord(c) > 127 for c in str(r.get('text', ''))),
+            'credit': lambda r: r.get('type') == 'credit',
+        }
+        
+        # Oversampling weights (higher = more samples)
+        self.oversample_weights = {
+            'failed_txn': 5.0,
+            'pending_txn': 3.0,
+            'large_amount': 2.0,
+            'small_amount': 2.0,
+            'unicode': 4.0,
+            'credit': 1.5,
+        }
+    
+    def identify_edge_cases(self, record: Dict) -> List[str]:
+        """Identify which edge cases a record matches."""
+        return [
+            name for name, condition in self.edge_cases.items()
+            if condition(record)
+        ]
+    
+    def calculate_sample_weight(self, record: Dict) -> float:
+        """Calculate importance weight for a record."""
+        weight = 1.0
+        for edge_case in self.identify_edge_cases(record):
+            weight *= self.oversample_weights.get(edge_case, 1.0)
+        return weight
+    
+    def oversample(
+        self,
+        data: List[Dict],
+        target_size: Optional[int] = None
+    ) -> List[Dict]:
+        """
+        Oversample data with edge case weighting.
+        
+        Returns dataset with increased representation of rare cases.
+        """
+        if target_size is None:
+            target_size = len(data)
+        
+        # Calculate weights
+        weights = [self.calculate_sample_weight(r) for r in data]
+        total_weight = sum(weights)
+        probs = [w / total_weight for w in weights]
+        
+        # Sample with replacement
+        indices = self.rng.choices(range(len(data)), weights=probs, k=target_size)
+        
+        oversampled = []
+        for i in indices:
+            record = data[i].copy()
+            record['oversampled'] = True
+            oversampled.append(record)
+        
+        return oversampled
+    
+    def generate_targeted_edge_cases(
+        self,
+        generator,
+        edge_case_type: str,
+        count: int
+    ) -> List[Dict]:
+        """Generate specific edge case samples."""
+        samples = []
+        
+        if edge_case_type == 'failed_txn':
+            from scripts.data_pipeline.generate_synthetic import TransactionStatus
+            for _ in range(count):
+                sample = generator.generate_transaction(
+                    status=TransactionStatus.FAILED
+                )
+                samples.append(sample)
+        
+        elif edge_case_type == 'large_amount':
+            for _ in range(count):
+                sample = generator.generate_transaction()
+                # Force large amount
+                sample['ground_truth']['amount'] = self.rng.uniform(100000, 1000000)
+                samples.append(sample)
+        
+        elif edge_case_type == 'unicode':
+            for _ in range(count):
+                sample = generator.generate_transaction()
+                # Use Hindi name
+                sample['ground_truth']['beneficiary'] = self.rng.choice([
+                    "राहुल शर्मा", "प्रिया सिंह", "అమిత్ కుమార్"
+                ])
+                samples.append(sample)
+        
+        return samples
+
+
+# ============================================================================
+# DOCUMENT/PDF GENERATION (Placeholder - needs external libs)
+# ============================================================================
+
+class DocumentGenerator:
+    """
+    Generate synthetic bank statements and documents.
+    
+    Note: Full implementation requires:
+    - reportlab for PDF generation
+    - PIL for image processing
+    - wkhtmltopdf for HTML to PDF
+    """
+    
+    STATEMENT_TEMPLATE = """
+    ============================================
+    {bank} BANK
+    ACCOUNT STATEMENT
+    ============================================
+    
+    Account Holder: {name}
+    Account Number: XXXXXXXX{account}
+    Statement Period: {start_date} to {end_date}
+    
+    Opening Balance: Rs. {opening_balance}
+    
+    --------------------------------------------
+    Date        Description          Debit    Credit    Balance
+    --------------------------------------------
+    {transactions}
+    --------------------------------------------
+    
+    Closing Balance: Rs. {closing_balance}
+    
+    This is a computer-generated statement.
+    """
+    
+    @classmethod
+    def generate_text_statement(
+        cls,
+        transactions: List[Dict],
+        bank: str,
+        account: str,
+        name: str,
+        rng: random.Random
+    ) -> str:
+        """Generate a text-based bank statement."""
+        if not transactions:
+            return ""
+        
+        # Sort by date
+        sorted_txns = sorted(
+            transactions,
+            key=lambda x: x.get('date', '2025-01-01')
+        )
+        
+        # Calculate running balance
+        opening = rng.randint(10000, 100000)
+        balance = opening
+        lines = []
+        
+        for txn in sorted_txns:
+            amount = txn.get('amount', 0)
+            txn_type = txn.get('type', 'debit')
+            
+            if txn_type == 'debit':
+                balance -= amount
+                debit = f"{amount:,.2f}"
+                credit = ""
+            else:
+                balance += amount
+                debit = ""
+                credit = f"{amount:,.2f}"
+            
+            desc = txn.get('merchant') or txn.get('beneficiary') or 'Transaction'
+            date_str = txn.get('date', '2025-01-01')
+            
+            line = f"{date_str}  {desc[:20]:<20}  {debit:>10}  {credit:>10}  {balance:>12,.2f}"
+            lines.append(line)
+        
+        start_date = sorted_txns[0].get('date', '2025-01-01')
+        end_date = sorted_txns[-1].get('date', '2025-01-31')
+        
+        return cls.STATEMENT_TEMPLATE.format(
+            bank=bank,
+            name=name,
+            account=account[-4:],
+            start_date=start_date,
+            end_date=end_date,
+            opening_balance=f"{opening:,.2f}",
+            closing_balance=f"{balance:,.2f}",
+            transactions='\n    '.join(lines)
+        )
+    
+    @classmethod
+    def generate_statement_image_data(
+        cls,
+        transactions: List[Dict],
+        bank: str,
+        rng: random.Random
+    ) -> Dict:
+        """
+        Generate data for statement image (actual rendering needs PIL).
+        
+        Returns structured data that can be used with image generation.
+        """
+        return {
+            'type': 'bank_statement',
+            'bank': bank,
+            'transactions': transactions,
+            'format': 'image_data',
+            'note': 'Use PIL/reportlab to render actual image'
+        }
+
+
+# ============================================================================
+# UNIFIED ADVANCED GENERATOR
+# ============================================================================
+
+class AdvancedSyntheticGenerator:
+    """
+    Unified generator combining all advanced features.
+    
+    Features:
+    1. Markov chain learning from real data
+    2. Statistical calibration
+    3. Multilingual support
+    4. Data augmentation
+    5. Edge case oversampling
+    """
+    
+    def __init__(self, seed: int = 42):
+        self.seed = seed
+        self.rng = random.Random(seed)
+        
+        # Components
+        self.markov: Optional[MarkovChain] = None
+        self.calibrator: Optional[DataCalibrator] = None
+        self.augmenter = DataAugmenter(seed)
+        self.edge_sampler = RareEdgeCaseSampler(seed)
+    
+    def train_on_real_data(self, real_data: List[Dict]):
+        """Train/calibrate on real data."""
+        print("Training on real data...")
+        
+        # Train Markov chain
+        texts = [r.get('text') or r.get('input', '') for r in real_data]
+        self.markov = MarkovChain(order=2)
+        self.markov.train(texts)
+        print(f"  Markov chain trained on {len(texts)} samples")
+        
+        # Calibrate distributions
+        self.calibrator = DataCalibrator()
+        parsed_data = []
+        for r in real_data:
+            gt = r.get('ground_truth') or r.get('output', {})
+            if isinstance(gt, str):
+                gt = json.loads(gt)
+            parsed_data.append(gt)
+        
+        self.calibrator.fit_from_data(parsed_data)
+        print("  Distributions calibrated")
+    
+    def generate(
+        self,
+        count: int,
+        languages: List[Language] = None,
+        include_documents: bool = False,
+        augmentation_factor: int = 1,
+        edge_case_ratio: float = 0.1,
+    ) -> List[Dict]:
+        """
+        Generate synthetic data with all advanced features.
+        
+        Args:
+            count: Number of records
+            languages: Languages to include (None = English only)
+            include_documents: Include bank statement format
+            augmentation_factor: How many augmented versions per sample
+            edge_case_ratio: Proportion of edge cases to include
+        """
+        if languages is None:
+            languages = [Language.ENGLISH]
+        
+        records = []
+        base_count = int(count / augmentation_factor)
+        edge_count = int(base_count * edge_case_ratio)
+        normal_count = base_count - edge_count
+        
+        print(f"Generating {count:,} records...")
+        print(f"  Base: {base_count:,}, Edges: {edge_count:,}, Augmented: {count - base_count:,}")
+        
+        # Generate normal transactions
+        for i in range(normal_count):
+            lang = self.rng.choice(languages)
+            
+            # Sample from calibrated distributions if available
+            if self.calibrator:
+                category = self.calibrator.sample_category(self.rng)
+                bank = self.calibrator.sample_bank(self.rng)
+                amount = self.calibrator.sample_amount(category, self.rng)
+            else:
+                category = self.rng.choice(['shopping', 'food', 'transfer', 'bills'])
+                bank = self.rng.choice(['HDFC', 'ICICI', 'SBI', 'Axis'])
+                amount = self.rng.uniform(100, 10000)
+            
+            # Generate message
+            is_debit = self.rng.random() < 0.7
+            template_type = 'debit' if is_debit else 'credit'
+            
+            params = {
+                'bank': bank,
+                'amount': f"{amount:,.2f}",
+                'account': str(self.rng.randint(1000, 9999)),
+                'date': (date.today() - timedelta(days=self.rng.randint(0, 365))).strftime('%d-%m-%Y'),
+                'vpa': f"{self.rng.choice(['swiggy', 'amazon', 'paytm'])}@ybl",
+                'sender': 'PhonePe',
+                'ref': ''.join(self.rng.choices('0123456789', k=12)),
+            }
+            
+            text = MultilingualBank.generate(template_type, lang, params)
+            
+            records.append({
+                'text': text,
+                'ground_truth': {
+                    'amount': round(amount, 2),
+                    'type': 'debit' if is_debit else 'credit',
+                    'bank': bank,
+                    'category': category,
+                    'language': lang.value,
+                },
+                'language': lang.value,
+            })
+            
+            if (i + 1) % 5000 == 0:
+                print(f"  Generated {i+1:,}/{base_count:,}")
+        
+        # Generate edge cases
+        for i in range(edge_count):
+            lang = self.rng.choice(languages)
+            edge_type = self.rng.choice(['unicode', 'large_amount', 'small_amount'])
+            
+            if edge_type == 'unicode' and lang == Language.ENGLISH:
+                lang = Language.HINDI
+            
+            amount = (
+                self.rng.uniform(100000, 1000000) if edge_type == 'large_amount'
+                else self.rng.uniform(0.5, 10) if edge_type == 'small_amount'
+                else self.rng.uniform(100, 10000)
+            )
+            
+            params = {
+                'bank': self.rng.choice(['HDFC', 'ICICI', 'SBI']),
+                'amount': f"{amount:,.2f}",
+                'account': str(self.rng.randint(1000, 9999)),
+                'date': date.today().strftime('%d-%m-%Y'),
+                'vpa': 'merchant@ybl',
+                'sender': MultilingualNameGenerator.get_name(lang, self.rng) if edge_type == 'unicode' else 'User',
+                'ref': ''.join(self.rng.choices('0123456789', k=12)),
+            }
+            
+            text = MultilingualBank.generate('debit', lang, params, use_native_numbers=(edge_type == 'unicode'))
+            
+            records.append({
+                'text': text,
+                'ground_truth': {
+                    'amount': round(amount, 2),
+                    'type': 'debit',
+                    'language': lang.value,
+                },
+                'edge_case': edge_type,
+                'language': lang.value,
+            })
+        
+        # Augment if factor > 1
+        if augmentation_factor > 1:
+            print(f"  Augmenting {len(records):,} records...")
+            records = self.augmenter.augment_batch(records, augmentation_factor)
+        
+        # Add statements if requested
+        if include_documents:
+            print("  Generating document samples...")
+            for _ in range(min(100, count // 100)):
+                bank = self.rng.choice(['HDFC', 'ICICI', 'SBI'])
+                account = str(self.rng.randint(10000000, 99999999))
+                name = self.rng.choice(['Rahul Sharma', 'Priya Singh', 'Amit Kumar'])
+                
+                # Use recent records for statement
+                txns = [r['ground_truth'] for r in self.rng.sample(records, min(10, len(records)))]
+                statement = DocumentGenerator.generate_text_statement(
+                    txns, bank, account, name, self.rng
+                )
+                
+                records.append({
+                    'text': statement,
+                    'ground_truth': {'document_type': 'bank_statement', 'bank': bank},
+                    'document': True,
+                })
+        
+        self.rng.shuffle(records)
+        
+        # Add IDs
+        for i, r in enumerate(records):
+            r['id'] = i + 1
+        
+        print(f"✅ Generated {len(records):,} total records")
+        return records
+    
+    def save_training_data(self, records: List[Dict], output_path: Path):
+        """Save in training format."""
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        
+        with open(output_path, 'w', encoding='utf-8') as f:
+            for r in records:
+                line = {
+                    'input': r['text'],
+                    'output': json.dumps(r['ground_truth'], ensure_ascii=False),
+                    'id': r.get('id'),
+                    'language': r.get('language', 'en'),
+                }
+                if r.get('edge_case'):
+                    line['edge_case'] = r['edge_case']
+                f.write(json.dumps(line, ensure_ascii=False) + '\n')
+        
+        print(f"✅ Saved to: {output_path}")
+
+
+# ============================================================================
+# CLI
+# ============================================================================
+
+def main():
+    parser = argparse.ArgumentParser(description="Advanced Synthetic Data Generator v4.0")
+    parser.add_argument("-n", "--count", type=int, default=10000, help="Number of records")
+    parser.add_argument("-o", "--output", default="data/synthetic/advanced_synthetic.jsonl")
+    parser.add_argument("--seed", type=int, default=42, help="Random seed")
+    parser.add_argument("--languages", nargs='+', default=['en'], 
+                       help="Languages: en, hi, ta, te, bn, kn, mr, gu")
+    parser.add_argument("--augment", type=int, default=1, help="Augmentation factor")
+    parser.add_argument("--edge-ratio", type=float, default=0.1, help="Edge case ratio")
+    parser.add_argument("--real-data", help="Path to real data for calibration")
+    parser.add_argument("--documents", action="store_true", help="Include document samples")
+    
+    args = parser.parse_args()
+    
+    # Parse languages
+    lang_map = {
+        'en': Language.ENGLISH, 'hi': Language.HINDI, 'ta': Language.TAMIL,
+        'te': Language.TELUGU, 'bn': Language.BENGALI, 'kn': Language.KANNADA,
+        'mr': Language.MARATHI, 'gu': Language.GUJARATI,
+    }
+    languages = [lang_map.get(l, Language.ENGLISH) for l in args.languages]
+    
+    # Initialize generator
+    generator = AdvancedSyntheticGenerator(seed=args.seed)
+    
+    # Train on real data if provided
+    if args.real_data:
+        real_path = Path(args.real_data)
+        if real_path.exists():
+            with open(real_path) as f:
+                real_data = [json.loads(line) for line in f]
+            generator.train_on_real_data(real_data)
+    
+    # Generate
+    records = generator.generate(
+        count=args.count,
+        languages=languages,
+        include_documents=args.documents,
+        augmentation_factor=args.augment,
+        edge_case_ratio=args.edge_ratio,
+    )
+    
+    # Save
+    output_path = Path(args.output)
+    generator.save_training_data(records, output_path)
+    
+    # Summary
+    print("\n📊 Summary:")
+    lang_counts = Counter(r.get('language', 'en') for r in records)
+    for lang, count in lang_counts.most_common():
+        print(f"  {lang}: {count:,}")
+    
+    edge_counts = Counter(r.get('edge_case') for r in records if r.get('edge_case'))
+    if edge_counts:
+        print("\n📋 Edge Cases:")
+        for edge, count in edge_counts.most_common():
+            print(f"  {edge}: {count:,}")
+
+
+if __name__ == "__main__":
+    main()

scripts/data_pipeline/generate_synthetic.py

@@ -0,0 +1,1726 @@
+#!/usr/bin/env python3
+"""
+Production-Grade Synthetic Data Generator for Indian Banking Transactions
+==========================================================================
+
+Engineering Principles:
+1. Grammar-based message generation (not hardcoded templates)
+2. Combinatorial coverage with configurable sampling
+3. Type-safe generation with validation
+4. Systematic edge case enumeration
+5. Statistical distribution control
+6. Reproducible with proper seeding
+7. Property-based validation
+
+Author: Ranjit Behera
+Version: 3.0 (Engineering Grade)
+"""
+
+from __future__ import annotations
+import json
+import random
+import hashlib
+import argparse
+import math
+import itertools
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field, asdict
+from datetime import datetime, timedelta, date
+from decimal import Decimal, ROUND_HALF_UP
+from enum import Enum, auto
+from pathlib import Path
+from typing import (
+    List, Dict, Optional, Tuple, Set, Generator, 
+    Callable, TypeVar, Generic, Union, Any, Iterator
+)
+from collections import defaultdict
+import re
+
+
+# ============================================================================
+# TYPE SYSTEM - Enums and Value Objects
+# ============================================================================
+
+class TransactionType(Enum):
+    DEBIT = "debit"
+    CREDIT = "credit"
+
+class TransactionStatus(Enum):
+    SUCCESS = "success"
+    FAILED = "failed"
+    PENDING = "pending"
+    REVERSED = "reversed"
+
+class PaymentMethod(Enum):
+    UPI = "upi"
+    NEFT = "neft"
+    RTGS = "rtgs"
+    IMPS = "imps"
+    CREDIT_CARD = "credit_card"
+    DEBIT_CARD = "debit_card"
+    ATM = "atm"
+    WALLET = "wallet"
+    AUTO_DEBIT = "auto_debit"
+    CHEQUE = "cheque"
+    CASH = "cash"
+
+class Category(Enum):
+    FOOD = "food"
+    GROCERY = "grocery"
+    SHOPPING = "shopping"
+    TRANSPORT = "transport"
+    TRAVEL = "travel"
+    FUEL = "fuel"
+    BILLS = "bills"
+    ENTERTAINMENT = "entertainment"
+    HEALTHCARE = "healthcare"
+    INVESTMENT = "investment"
+    INSURANCE = "insurance"
+    EDUCATION = "education"
+    TRANSFER = "transfer"
+    SALARY = "salary"
+    REFUND = "refund"
+    CASHBACK = "cashback"
+    EMI = "emi"
+    ATM_WITHDRAWAL = "atm_withdrawal"
+    OTHER = "other"
+
+class MessageType(Enum):
+    TRANSACTION = "transaction"
+    OTP = "otp"
+    PROMOTIONAL = "promotional"
+    ALERT = "alert"
+    STATEMENT = "statement"
+
+class AmountFormat(Enum):
+    """All possible amount format variations."""
+    INTEGER = "integer"                    # 2500
+    DECIMAL_2 = "decimal_2"                # 2500.00
+    DECIMAL_1 = "decimal_1"                # 2500.5
+    COMMA_INTERNATIONAL = "comma_intl"     # 2,500.00
+    COMMA_INDIAN = "comma_indian"          # 2,50,000.00
+    NO_DECIMAL_COMMA = "no_decimal_comma"  # 2,500
+    COMPACT = "compact"                    # 2.5K, 1.2L
+    PADDED = "padded"                      # 002500.00
+
+class DateFormat(Enum):
+    """All possible date format variations."""
+    DD_MM_YYYY_DASH = "dd-mm-yyyy"         # 28-12-2025
+    DD_MM_YY_DASH = "dd-mm-yy"             # 28-12-25
+    DD_MM_YYYY_SLASH = "dd/mm/yyyy"        # 28/12/2025
+    DD_MM_YY_SLASH = "dd/mm/yy"            # 28/12/25
+    DD_MON_YY = "dd-mon-yy"                # 28-Dec-25
+    DD_MON_YYYY = "dd-mon-yyyy"            # 28-Dec-2025
+    DD_MONTH_YYYY = "dd-month-yyyy"        # 28 December 2025
+    MON_DD_YYYY = "mon-dd-yyyy"            # Dec 28, 2025
+    YYYY_MM_DD = "yyyy-mm-dd"              # 2025-12-28 (ISO)
+    COMPACT = "compact"                    # 28Dec25
+    RELATIVE = "relative"                  # today, yesterday
+
+class CurrencySymbol(Enum):
+    """Currency symbol variations."""
+    RS_DOT = "Rs."
+    RS = "Rs"
+    INR = "INR"
+    RUPEE = "₹"
+    RS_SPACE = "Rs "
+    INR_SPACE = "INR "
+
+
+# ============================================================================
+# VALUE OBJECTS - Immutable domain objects
+# ============================================================================
+
+@dataclass(frozen=True)
+class Amount:
+    """Immutable amount with validation."""
+    value: Decimal
+    
+    def __post_init__(self):
+        if self.value < 0:
+            raise ValueError(f"Amount cannot be negative: {self.value}")
+    
+    @classmethod
+    def from_float(cls, value: float) -> 'Amount':
+        return cls(Decimal(str(value)).quantize(Decimal('0.01'), rounding=ROUND_HALF_UP))
+    
+    @classmethod
+    def from_int(cls, value: int) -> 'Amount':
+        return cls(Decimal(value))
+    
+    def format(self, fmt: AmountFormat) -> str:
+        """Format amount according to specified format."""
+        v = float(self.value)
+        
+        if fmt == AmountFormat.INTEGER:
+            return str(int(v))
+        elif fmt == AmountFormat.DECIMAL_2:
+            return f"{v:.2f}"
+        elif fmt == AmountFormat.DECIMAL_1:
+            return f"{v:.1f}" if v != int(v) else str(int(v))
+        elif fmt == AmountFormat.COMMA_INTERNATIONAL:
+            return f"{v:,.2f}"
+        elif fmt == AmountFormat.COMMA_INDIAN:
+            return self._indian_format(v)
+        elif fmt == AmountFormat.NO_DECIMAL_COMMA:
+            return f"{int(v):,}"
+        elif fmt == AmountFormat.COMPACT:
+            return self._compact_format(v)
+        elif fmt == AmountFormat.PADDED:
+            return f"{v:012.2f}"
+        return str(v)
+    
+    def _indian_format(self, v: float) -> str:
+        """Format in Indian numbering system (lakhs, crores)."""
+        s = f"{v:.2f}"
+        integer_part, decimal_part = s.split('.')
+        
+        if len(integer_part) <= 3:
+            return s
+        
+        # Last 3 digits
+        result = integer_part[-3:]
+        integer_part = integer_part[:-3]
+        
+        # Rest in groups of 2
+        while integer_part:
+            result = integer_part[-2:] + ',' + result
+            integer_part = integer_part[:-2]
+        
+        return result + '.' + decimal_part
+    
+    def _compact_format(self, v: float) -> str:
+        """Format as compact (2.5K, 1.2L, 3.5Cr)."""
+        if v >= 10000000:  # Crore
+            return f"{v/10000000:.1f}Cr"
+        elif v >= 100000:  # Lakh
+            return f"{v/100000:.1f}L"
+        elif v >= 1000:  # Thousand
+            return f"{v/1000:.1f}K"
+        return str(int(v))
+
+
+@dataclass(frozen=True)
+class Account:
+    """Bank account representation."""
+    last_four: str
+    
+    def __post_init__(self):
+        if not self.last_four.isdigit() or len(self.last_four) != 4:
+            raise ValueError(f"Invalid account last four: {self.last_four}")
+    
+    def format(self, style: str = "XX") -> str:
+        """Format account number."""
+        styles = {
+            "XX": f"XX{self.last_four}",
+            "xx": f"xx{self.last_four}",
+            "****": f"****{self.last_four}",
+            "...": f"...{self.last_four}",
+            "plain": self.last_four,
+            "X": f"X{self.last_four}",
+        }
+        return styles.get(style, f"XX{self.last_four}")
+
+
+@dataclass(frozen=True)
+class Reference:
+    """Transaction reference number."""
+    value: str
+    ref_type: str  # numeric, alphanumeric, utr
+    
+    @classmethod
+    def generate(cls, ref_type: str, bank_code: str = "HDFC") -> 'Reference':
+        if ref_type == "numeric":
+            value = ''.join(random.choices('0123456789', k=12))
+        elif ref_type == "alphanumeric":
+            value = bank_code + ''.join(random.choices('0123456789', k=11))
+        elif ref_type == "utr":
+            date_part = datetime.now().strftime('%y%m%d')
+            value = f"{bank_code}{date_part}N{''.join(random.choices('0123456789', k=6))}"
+        else:
+            value = ''.join(random.choices('0123456789', k=12))
+        
+        return cls(value=value, ref_type=ref_type)
+
+
+@dataclass(frozen=True)
+class TransactionDate:
+    """Date with multiple format support."""
+    date: date
+    time: Optional[Tuple[int, int, int]] = None  # (hour, minute, second)
+    
+    def format_date(self, fmt: DateFormat) -> str:
+        """Format date according to specified format."""
+        d = self.date
+        
+        formats = {
+            DateFormat.DD_MM_YYYY_DASH: d.strftime("%d-%m-%Y"),
+            DateFormat.DD_MM_YY_DASH: d.strftime("%d-%m-%y"),
+            DateFormat.DD_MM_YYYY_SLASH: d.strftime("%d/%m/%Y"),
+            DateFormat.DD_MM_YY_SLASH: d.strftime("%d/%m/%y"),
+            DateFormat.DD_MON_YY: d.strftime("%d-%b-%y"),
+            DateFormat.DD_MON_YYYY: d.strftime("%d-%b-%Y"),
+            DateFormat.DD_MONTH_YYYY: d.strftime("%d %B %Y"),
+            DateFormat.MON_DD_YYYY: d.strftime("%b %d, %Y"),
+            DateFormat.YYYY_MM_DD: d.strftime("%Y-%m-%d"),
+            DateFormat.COMPACT: d.strftime("%d%b%y"),
+        }
+        
+        if fmt == DateFormat.RELATIVE:
+            today = date.today()
+            diff = (today - d).days
+            if diff == 0:
+                return "today"
+            elif diff == 1:
+                return "yesterday"
+            else:
+                return formats[DateFormat.DD_MM_YY_DASH]
+        
+        return formats.get(fmt, d.strftime("%d-%m-%Y"))
+    
+    def format_time(self) -> Optional[str]:
+        """Format time if present."""
+        if not self.time:
+            return None
+        h, m, s = self.time
+        formats = [
+            f"{h:02d}:{m:02d}:{s:02d}",
+            f"{h:02d}:{m:02d}",
+            f"{h:02d}:{m:02d} {'AM' if h < 12 else 'PM'}",
+        ]
+        return random.choice(formats)
+    
+    def normalized(self) -> str:
+        """Return ISO format."""
+        return self.date.strftime("%Y-%m-%d")
+
+
+# ============================================================================
+# GRAMMAR-BASED MESSAGE GENERATOR
+# ============================================================================
+
+class GrammarRule:
+    """Represents a grammar rule with weighted alternatives."""
+    
+    def __init__(self, name: str, alternatives: List[Tuple[str, float]]):
+        """
+        Args:
+            name: Rule name
+            alternatives: List of (pattern, weight) tuples
+        """
+        self.name = name
+        self.alternatives = alternatives
+        self._normalize_weights()
+    
+    def _normalize_weights(self):
+        """Normalize weights to sum to 1."""
+        total = sum(w for _, w in self.alternatives)
+        self.alternatives = [(p, w/total) for p, w in self.alternatives]
+    
+    def sample(self, rng: random.Random) -> str:
+        """Sample one alternative based on weights."""
+        r = rng.random()
+        cumulative = 0
+        for pattern, weight in self.alternatives:
+            cumulative += weight
+            if r <= cumulative:
+                return pattern
+        return self.alternatives[-1][0]
+
+
+class MessageGrammar:
+    """
+    Grammar-based message generator.
+    
+    Grammar Structure:
+    MESSAGE ::= PREFIX BODY SUFFIX
+    PREFIX  ::= BANK_INTRO
+    BODY    ::= AMOUNT_PHRASE ACCOUNT_PHRASE DATE_PHRASE PARTY_PHRASE REF_PHRASE
+    SUFFIX  ::= BALANCE_PHRASE? WARNING_PHRASE?
+    """
+    
+    def __init__(self, seed: int = 42):
+        self.rng = random.Random(seed)
+        self._build_grammar()
+    
+    def _build_grammar(self):
+        """Build the grammar rules."""
+        
+        # Bank introduction patterns
+        self.bank_intro = GrammarRule("BANK_INTRO", [
+            ("{bank}:", 0.3),
+            ("{bank} Bank:", 0.25),
+            ("{bank} Bk:", 0.1),
+            ("Dear Customer, {bank}:", 0.1),
+            ("Alert: {bank}", 0.1),
+            ("{bank} Bank Alert:", 0.1),
+            ("{bank} Bank Acct", 0.05),
+        ])
+        
+        # Amount phrase patterns for DEBIT
+        self.amount_debit = GrammarRule("AMOUNT_DEBIT", [
+            ("{currency}{amount} debited from", 0.25),
+            ("{currency}{amount} sent to", 0.15),
+            ("{currency}{amount} paid to", 0.1),
+            ("{currency}{amount} transferred to", 0.1),
+            ("{currency} {amount} debited from", 0.1),
+            ("INR {amount} debited from", 0.1),
+            ("{currency}{amount} spent at", 0.1),
+            ("Amount {currency}{amount} debited", 0.1),
+        ])
+        
+        # Amount phrase patterns for CREDIT
+        self.amount_credit = GrammarRule("AMOUNT_CREDIT", [
+            ("{currency}{amount} credited to", 0.3),
+            ("{currency}{amount} received in", 0.2),
+            ("{currency}{amount} deposited to", 0.1),
+            ("{currency} {amount} credited to", 0.15),
+            ("INR {amount} credited to", 0.1),
+            ("{currency}{amount} added to", 0.1),
+            ("Received {currency}{amount} in", 0.05),
+        ])
+        
+        # Account phrase patterns
+        self.account_phrase = GrammarRule("ACCOUNT_PHRASE", [
+            ("A/c {account}", 0.3),
+            ("a/c {account}", 0.15),
+            ("Account {account}", 0.15),
+            ("Acct {account}", 0.15),
+            ("A/c No. {account}", 0.1),
+            ("Acc {account}", 0.1),
+            ("your A/c {account}", 0.05),
+        ])
+        
+        # Date phrase patterns
+        self.date_phrase = GrammarRule("DATE_PHRASE", [
+            ("on {date}", 0.4),
+            ("on {date} {time}", 0.2),
+            ("{date}", 0.15),
+            ("dated {date}", 0.1),
+            ("on {date} at {time}", 0.1),
+            ("{date} {time}", 0.05),
+        ])
+        
+        # Counterparty patterns (UPI)
+        self.party_upi = GrammarRule("PARTY_UPI", [
+            ("VPA {vpa}", 0.2),
+            ("to {vpa}", 0.2),
+            ("VPA: {vpa}", 0.15),
+            ("to VPA {vpa}", 0.15),
+            ("Info: UPI/{vpa}", 0.1),
+            ("{beneficiary} ({vpa})", 0.1),
+            ("UPI-{vpa}", 0.1),
+        ])
+        
+        # Reference patterns
+        self.reference = GrammarRule("REFERENCE", [
+            ("Ref: {ref}", 0.25),
+            ("Ref {ref}", 0.2),
+            ("UPI Ref: {ref}", 0.2),
+            ("Reference: {ref}", 0.1),
+            ("UTR: {ref}", 0.1),
+            ("Txn Ref: {ref}", 0.1),
+            ("Ref No. {ref}", 0.05),
+        ])
+        
+        # Balance patterns
+        self.balance = GrammarRule("BALANCE", [
+            ("Avl Bal: {currency}{balance}", 0.25),
+            ("Bal: {currency}{balance}", 0.2),
+            ("Available Balance: {currency}{balance}", 0.15),
+            ("Avl Bal {currency}{balance}", 0.15),
+            ("Balance: {currency}{balance}", 0.1),
+            ("A/c Bal: {currency}{balance}", 0.1),
+            ("", 0.05),  # No balance shown
+        ])
+        
+        # Warning patterns
+        self.warning = GrammarRule("WARNING", [
+            ("Not you? Call {helpline}", 0.3),
+            ("If not done by you, call {helpline}", 0.2),
+            ("Call {helpline} for dispute", 0.2),
+            ("", 0.3),  # No warning
+        ])
+        
+        # Failed transaction patterns
+        self.failed = GrammarRule("FAILED", [
+            ("Transaction of {currency}{amount} FAILED", 0.3),
+            ("{currency}{amount} transaction DECLINED", 0.25),
+            ("Payment of {currency}{amount} FAILED", 0.2),
+            ("Transaction FAILED: {currency}{amount}", 0.15),
+            ("{currency}{amount} to {vpa} DECLINED", 0.1),
+        ])
+        
+        # Pending patterns
+        self.pending = GrammarRule("PENDING", [
+            ("{currency}{amount} debit PENDING", 0.35),
+            ("Transaction of {currency}{amount} is PENDING", 0.3),
+            ("{currency}{amount} payment is being processed", 0.2),
+            ("Processing: {currency}{amount} to {vpa}", 0.15),
+        ])
+    
+    def generate_debit_message(self, params: Dict[str, str]) -> str:
+        """Generate a debit transaction message."""
+        parts = [
+            self.bank_intro.sample(self.rng),
+            self.amount_debit.sample(self.rng),
+            self.account_phrase.sample(self.rng),
+            self.date_phrase.sample(self.rng),
+            self.party_upi.sample(self.rng) if params.get('vpa') else "",
+            self.reference.sample(self.rng),
+            self.balance.sample(self.rng),
+            self.warning.sample(self.rng),
+        ]
+        
+        message = " ".join(p for p in parts if p)
+        return self._apply_params(message, params)
+    
+    def generate_credit_message(self, params: Dict[str, str]) -> str:
+        """Generate a credit transaction message."""
+        parts = [
+            self.bank_intro.sample(self.rng),
+            self.amount_credit.sample(self.rng),
+            self.account_phrase.sample(self.rng),
+            self.date_phrase.sample(self.rng),
+            f"from {params.get('vpa', params.get('beneficiary', ''))}",
+            self.reference.sample(self.rng),
+            self.balance.sample(self.rng),
+        ]
+        
+        message = " ".join(p for p in parts if p)
+        return self._apply_params(message, params)
+    
+    def generate_failed_message(self, params: Dict[str, str]) -> str:
+        """Generate a failed transaction message."""
+        parts = [
+            self.bank_intro.sample(self.rng),
+            self.failed.sample(self.rng),
+            self.account_phrase.sample(self.rng),
+            f"Reason: {params.get('reason', 'Transaction declined')}",
+            self.reference.sample(self.rng),
+        ]
+        
+        message = " ".join(p for p in parts if p)
+        return self._apply_params(message, params)
+    
+    def generate_pending_message(self, params: Dict[str, str]) -> str:
+        """Generate a pending transaction message."""
+        parts = [
+            self.bank_intro.sample(self.rng),
+            self.pending.sample(self.rng),
+            f"for {params.get('vpa', '')}",
+            self.account_phrase.sample(self.rng),
+            self.reference.sample(self.rng),
+        ]
+        
+        message = " ".join(p for p in parts if p)
+        return self._apply_params(message, params)
+    
+    def _apply_params(self, template: str, params: Dict[str, str]) -> str:
+        """Apply parameters to template."""
+        for key, value in params.items():
+            template = template.replace(f"{{{key}}}", str(value) if value else "")
+        
+        # Clean up double spaces and trailing punctuation
+        template = re.sub(r'\s+', ' ', template)
+        template = re.sub(r'\s+([.,:])', r'\1', template)
+        return template.strip()
+
+
+# ============================================================================
+# COMBINATORIAL GENERATOR
+# ============================================================================
+
+@dataclass
+class GenerationSpace:
+    """
+    Defines the combinatorial space for generation.
+    
+    Total combinations = product of all dimension sizes
+    """
+    banks: List[str]
+    amount_formats: List[AmountFormat]
+    date_formats: List[DateFormat]
+    currency_symbols: List[CurrencySymbol]
+    account_styles: List[str]
+    reference_types: List[str]
+    transaction_types: List[TransactionType]
+    statuses: List[TransactionStatus]
+    payment_methods: List[PaymentMethod]
+    categories: List[Category]
+    noise_levels: List[float]
+    
+    @property
+    def total_combinations(self) -> int:
+        """Calculate total theoretical combinations."""
+        return (
+            len(self.banks) *
+            len(self.amount_formats) *
+            len(self.date_formats) *
+            len(self.currency_symbols) *
+            len(self.account_styles) *
+            len(self.reference_types) *
+            len(self.transaction_types) *
+            len(self.statuses) *
+            len(self.payment_methods) *
+            len(self.categories) *
+            len(self.noise_levels)
+        )
+    
+    def get_dimension_coverage(self, samples: int) -> Dict[str, float]:
+        """Calculate coverage for each dimension given sample count."""
+        dimensions = {
+            'banks': len(self.banks),
+            'amount_formats': len(self.amount_formats),
+            'date_formats': len(self.date_formats),
+            'currency_symbols': len(self.currency_symbols),
+            'account_styles': len(self.account_styles),
+            'reference_types': len(self.reference_types),
+            'transaction_types': len(self.transaction_types),
+            'statuses': len(self.statuses),
+            'payment_methods': len(self.payment_methods),
+            'categories': len(self.categories),
+            'noise_levels': len(self.noise_levels),
+        }
+        
+        # Expected coverage using coupon collector approximation
+        coverage = {}
+        for dim, size in dimensions.items():
+            # E[samples needed to see all] ≈ n * H_n where H_n is harmonic number
+            harmonic = sum(1/i for i in range(1, size + 1))
+            expected_needed = size * harmonic
+            coverage[dim] = min(1.0, samples / expected_needed)
+        
+        return coverage
+
+
+class EdgeCaseGenerator:
+    """
+    Systematic edge case generation using boundary value analysis.
+    
+    Categories:
+    1. Amount boundaries
+    2. Date boundaries  
+    3. String boundaries
+    4. Format edge cases
+    5. Invalid inputs (for negative testing)
+    """
+    
+    # Amount edge cases
+    AMOUNT_BOUNDARIES = [
+        0.01,           # Minimum meaningful
+        0.50,           # Sub-rupee
+        0.99,           # Just under 1
+        1.00,           # Boundary
+        9.99,           # Just under 10
+        10.00,          # Boundary
+        99.99,          # Just under 100
+        100.00,         # Common boundary
+        499.99,         # Just under 500
+        500.00,         # Common payment
+        999.99,         # Just under 1000
+        1000.00,        # Thousand boundary
+        9999.99,        # Just under 10K
+        10000.00,       # 10K boundary
+        99999.99,       # Just under 1L
+        100000.00,      # Lakh boundary
+        999999.99,      # Just under 10L
+        1000000.00,     # 10L boundary
+        9999999.99,     # Just under 1Cr
+        10000000.00,    # Crore boundary
+    ]
+    
+    # Round numbers (common in real transactions)
+    ROUND_AMOUNTS = [
+        50, 100, 200, 250, 500, 750, 1000, 1500, 2000, 2500,
+        3000, 4000, 5000, 7500, 10000, 15000, 20000, 25000,
+        50000, 75000, 100000, 200000, 500000,
+    ]
+    
+    # Psychological pricing
+    PSYCHOLOGICAL_AMOUNTS = [
+        49, 99, 149, 199, 249, 299, 399, 499, 599, 699, 799, 899, 999,
+        1299, 1499, 1999, 2499, 2999, 3999, 4999, 9999,
+    ]
+    
+    # Date edge cases
+    DATE_EDGE_CASES = [
+        date(2024, 2, 29),   # Leap year
+        date(2025, 2, 28),   # Non-leap year Feb end
+        date(2025, 12, 31),  # Year end
+        date(2026, 1, 1),    # Year start
+        date(2025, 3, 31),   # Month with 31 days
+        date(2025, 4, 30),   # Month with 30 days
+    ]
+    
+    # Special characters in merchant names
+    SPECIAL_CHAR_MERCHANTS = [
+        "McDonald's",
+        "H&M",
+        "AT&T",
+        "L'Oreal",
+        "7-Eleven",
+        "Marks & Spencer",
+        "Johnson & Johnson",
+        "Dunkin' Donuts",
+        "Toys \"R\" Us",
+        "Yahoo!",
+    ]
+    
+    # Long merchant names
+    LONG_MERCHANTS = [
+        "Tata Consultancy Services Limited",
+        "Hindustan Petroleum Corporation Limited",
+        "Life Insurance Corporation of India",
+        "Indian Oil Corporation Limited",
+        "Bharat Heavy Electricals Limited",
+        "Steel Authority of India Limited",
+    ]
+    
+    # Unicode names (Hindi/Regional)
+    UNICODE_NAMES = [
+        "राहुल शर्मा",
+        "प्रिया सिंह",
+        "अमित कुमार",
+        "सुनीता देवी",
+        "విజయ్ కుమార్",
+        "প্রিয়া দাস",
+        "அருண் குமார்",
+    ]
+    
+    @classmethod
+    def get_amount_edge_cases(cls) -> List[float]:
+        """Get all amount edge cases."""
+        return cls.AMOUNT_BOUNDARIES + cls.ROUND_AMOUNTS + cls.PSYCHOLOGICAL_AMOUNTS
+    
+    @classmethod
+    def get_date_edge_cases(cls) -> List[date]:
+        """Get all date edge cases."""
+        today = date.today()
+        return cls.DATE_EDGE_CASES + [
+            today,
+            today - timedelta(days=1),
+            today - timedelta(days=7),
+            today - timedelta(days=30),
+            today - timedelta(days=365),
+        ]
+    
+    @classmethod
+    def generate_edge_case_batch(cls, count: int, rng: random.Random) -> List[Dict]:
+        """Generate a batch of edge case configurations."""
+        configs = []
+        
+        # Amount edge cases
+        for amount in rng.sample(cls.AMOUNT_BOUNDARIES, min(count // 4, len(cls.AMOUNT_BOUNDARIES))):
+            configs.append({'edge_type': 'amount_boundary', 'amount': amount})
+        
+        # Special character merchants
+        for merchant in cls.SPECIAL_CHAR_MERCHANTS:
+            configs.append({'edge_type': 'special_char', 'merchant': merchant})
+        
+        # Long merchants
+        for merchant in cls.LONG_MERCHANTS:
+            configs.append({'edge_type': 'long_merchant', 'merchant': merchant})
+        
+        # Unicode names
+        for name in cls.UNICODE_NAMES:
+            configs.append({'edge_type': 'unicode', 'beneficiary': name})
+        
+        return configs[:count]
+
+
+# ============================================================================
+# STATISTICAL DISTRIBUTION CONTROLLER
+# ============================================================================
+
+class DistributionConfig:
+    """
+    Configurable statistical distributions for realistic data.
+    
+    Uses weighted random selection based on real-world frequencies.
+    """
+    
+    # Transaction type distribution
+    TRANSACTION_TYPE_WEIGHTS = {
+        TransactionType.DEBIT: 0.7,
+        TransactionType.CREDIT: 0.3,
+    }
+    
+    # Status distribution
+    STATUS_WEIGHTS = {
+        TransactionStatus.SUCCESS: 0.92,
+        TransactionStatus.FAILED: 0.05,
+        TransactionStatus.PENDING: 0.02,
+        TransactionStatus.REVERSED: 0.01,
+    }
+    
+    # Payment method distribution
+    PAYMENT_METHOD_WEIGHTS = {
+        PaymentMethod.UPI: 0.55,
+        PaymentMethod.CREDIT_CARD: 0.15,
+        PaymentMethod.DEBIT_CARD: 0.08,
+        PaymentMethod.NEFT: 0.08,
+        PaymentMethod.IMPS: 0.05,
+        PaymentMethod.ATM: 0.04,
+        PaymentMethod.AUTO_DEBIT: 0.03,
+        PaymentMethod.WALLET: 0.02,
+    }
+    
+    # Category distribution (based on typical spending)
+    CATEGORY_WEIGHTS = {
+        Category.FOOD: 0.18,
+        Category.SHOPPING: 0.15,
+        Category.GROCERY: 0.12,
+        Category.TRANSFER: 0.12,
+        Category.BILLS: 0.10,
+        Category.TRANSPORT: 0.08,
+        Category.ENTERTAINMENT: 0.05,
+        Category.FUEL: 0.05,
+        Category.INVESTMENT: 0.04,
+        Category.HEALTHCARE: 0.03,
+        Category.TRAVEL: 0.03,
+        Category.SALARY: 0.02,
+        Category.EMI: 0.02,
+        Category.REFUND: 0.01,
+    }
+    
+    # Amount distribution by category (min, max, mean, std_dev)
+    AMOUNT_DISTRIBUTIONS = {
+        Category.FOOD: (20, 3000, 350, 300),
+        Category.GROCERY: (50, 10000, 800, 600),
+        Category.SHOPPING: (100, 100000, 2500, 3000),
+        Category.TRANSPORT: (20, 5000, 250, 300),
+        Category.TRAVEL: (500, 200000, 8000, 15000),
+        Category.FUEL: (100, 10000, 1500, 1000),
+        Category.BILLS: (100, 20000, 1200, 1500),
+        Category.ENTERTAINMENT: (50, 5000, 500, 400),
+        Category.HEALTHCARE: (100, 50000, 2000, 3000),
+        Category.INVESTMENT: (500, 500000, 25000, 50000),
+        Category.TRANSFER: (100, 200000, 5000, 10000),
+        Category.SALARY: (15000, 500000, 60000, 40000),
+        Category.EMI: (1000, 100000, 15000, 12000),
+        Category.REFUND: (50, 50000, 1000, 2000),
+    }
+    
+    @classmethod
+    def sample_weighted(cls, weights: Dict, rng: random.Random):
+        """Sample from weighted distribution."""
+        items = list(weights.keys())
+        probs = list(weights.values())
+        total = sum(probs)
+        probs = [p / total for p in probs]
+        
+        r = rng.random()
+        cumulative = 0
+        for item, prob in zip(items, probs):
+            cumulative += prob
+            if r <= cumulative:
+                return item
+        return items[-1]
+    
+    @classmethod
+    def sample_amount(cls, category: Category, rng: random.Random) -> float:
+        """Sample amount based on category distribution."""
+        min_amt, max_amt, mean, std = cls.AMOUNT_DISTRIBUTIONS.get(
+            category, (100, 10000, 1000, 1000)
+        )
+        
+        # Use truncated normal distribution
+        amount = rng.gauss(mean, std)
+        amount = max(min_amt, min(max_amt, amount))
+        
+        # Round to realistic precision
+        if amount < 100:
+            amount = round(amount, 2)
+        elif amount < 1000:
+            amount = round(amount, 1)
+        else:
+            amount = round(amount, 0)
+        
+        return amount
+
+
+# ============================================================================
+# NOISE INJECTION ENGINE
+# ============================================================================
+
+class NoiseEngine:
+    """
+    Systematic noise injection for realistic messages.
+    
+    Noise types:
+    1. Spacing variations
+    2. Abbreviations
+    3. Case changes
+    4. Truncation
+    5. Typos
+    6. Punctuation variations
+    """
+    
+    ABBREVIATIONS = {
+        'Account': ['A/c', 'Acc', 'Acct', 'a/c'],
+        'Reference': ['Ref', 'Ref.', 'ref'],
+        'Transaction': ['Txn', 'txn', 'Trans'],
+        'Available': ['Avl', 'Avail', 'avl'],
+        'Balance': ['Bal', 'bal', 'Bal.'],
+        'Credited': ['Cr', 'cr', 'Cr.'],
+        'Debited': ['Dr', 'dr', 'Dr.'],
+        'Number': ['No', 'No.', 'Num'],
+    }
+    
+    COMMON_TYPOS = {
+        'debited': ['debitd', 'debite', 'debitted'],
+        'credited': ['creditd', 'credite', 'creditted'],
+        'transaction': ['transction', 'transcation', 'transacton'],
+        'available': ['availble', 'avialable', 'availabel'],
+        'balance': ['balace', 'balnce', 'balanc'],
+    }
+    
+    def __init__(self, seed: int = 42):
+        self.rng = random.Random(seed)
+    
+    def apply(self, text: str, noise_level: float) -> str:
+        """
+        Apply noise to text.
+        
+        Args:
+            text: Original text
+            noise_level: 0.0 (clean) to 1.0 (very noisy)
+        """
+        if noise_level <= 0:
+            return text
+        
+        # Determine which noise types to apply
+        noise_budget = noise_level
+        
+        # 1. Abbreviations (always some)
+        if self.rng.random() < noise_budget * 0.5:
+            text = self._apply_abbreviations(text)
+        
+        # 2. Spacing
+        if self.rng.random() < noise_budget * 0.3:
+            text = self._vary_spacing(text)
+        
+        # 3. Case changes
+        if self.rng.random() < noise_budget * 0.2:
+            text = self._vary_case(text)
+        
+        # 4. Truncation (for SMS)
+        if self.rng.random() < noise_budget * 0.1:
+            text = self._truncate(text)
+        
+        # 5. Typos (rare)
+        if self.rng.random() < noise_budget * 0.05:
+            text = self._add_typos(text)
+        
+        # 6. Punctuation
+        if self.rng.random() < noise_budget * 0.2:
+            text = self._vary_punctuation(text)
+        
+        return text
+    
+    def _apply_abbreviations(self, text: str) -> str:
+        """Replace words with abbreviations."""
+        for word, abbrevs in self.ABBREVIATIONS.items():
+            if word in text and self.rng.random() < 0.5:
+                text = text.replace(word, self.rng.choice(abbrevs), 1)
+        return text
+    
+    def _vary_spacing(self, text: str) -> str:
+        """Vary spacing."""
+        variations = [
+            (". ", "."),
+            (": ", ":"),
+            ("Rs. ", "Rs."),
+            ("Rs ", "Rs"),
+            (", ", ","),
+        ]
+        for old, new in variations:
+            if self.rng.random() < 0.3:
+                text = text.replace(old, new)
+        return text
+    
+    def _vary_case(self, text: str) -> str:
+        """Vary case."""
+        if self.rng.random() < 0.3:
+            return text.upper()
+        elif self.rng.random() < 0.1:
+            return text.lower()
+        return text
+    
+    def _truncate(self, text: str) -> str:
+        """Truncate to SMS limit."""
+        if len(text) > 160:
+            return text[:157] + "..."
+        return text
+    
+    def _add_typos(self, text: str) -> str:
+        """Add occasional typos."""
+        for word, typos in self.COMMON_TYPOS.items():
+            if word in text.lower() and self.rng.random() < 0.2:
+                # Case-insensitive replace
+                pattern = re.compile(re.escape(word), re.IGNORECASE)
+                text = pattern.sub(self.rng.choice(typos), text, count=1)
+        return text
+    
+    def _vary_punctuation(self, text: str) -> str:
+        """Vary punctuation."""
+        if self.rng.random() < 0.3:
+            text = text.replace(".", "")
+        if self.rng.random() < 0.2:
+            text = text.replace(",", "")
+        return text
+
+
+# ============================================================================
+# MERCHANT & COUNTERPARTY DATABASE
+# ============================================================================
+
+@dataclass
+class Merchant:
+    """Merchant entity with VPA and metadata."""
+    name: str
+    vpa: str
+    category: Category
+    aliases: List[str] = field(default_factory=list)
+    
+    def get_display_name(self, rng: random.Random) -> str:
+        """Get a random display name variation."""
+        options = [self.name] + self.aliases
+        return rng.choice(options)
+
+
+class MerchantDatabase:
+    """
+    Database of merchants organized by category.
+    
+    Provides O(1) lookup by category and O(1) random selection.
+    """
+    
+    def __init__(self):
+        self._by_category: Dict[Category, List[Merchant]] = defaultdict(list)
+        self._by_name: Dict[str, Merchant] = {}
+        self._all: List[Merchant] = []
+        self._build_database()
+    
+    def _build_database(self):
+        """Build the merchant database."""
+        merchants_data = [
+            # Food Delivery
+            ("Swiggy", "swiggy@ybl", Category.FOOD, ["SWIGGY", "Swiggy Instamart"]),
+            ("Zomato", "zomato@paytm", Category.FOOD, ["ZOMATO", "Zomato Gold"]),
+            ("Dominos", "dominos@hdfcbank", Category.FOOD, ["DOMINOS", "Domino's Pizza"]),
+            ("Pizza Hut", "pizzahut@icici", Category.FOOD, ["PIZZA HUT"]),
+            ("McDonalds", "mcdonalds@ybl", Category.FOOD, ["McDonald's", "MCD"]),
+            ("KFC", "kfc@paytm", Category.FOOD, ["KFC"]),
+            ("Starbucks", "starbucks@icici", Category.FOOD, ["STARBUCKS"]),
+            ("Subway", "subway@hdfcbank", Category.FOOD, ["SUBWAY"]),
+            
+            # E-commerce
+            ("Amazon", "amazon@apl", Category.SHOPPING, ["AMAZON", "Amazon.in"]),
+            ("Flipkart", "flipkart@ybl", Category.SHOPPING, ["FLIPKART"]),
+            ("Myntra", "myntra@ybl", Category.SHOPPING, ["MYNTRA"]),
+            ("Ajio", "ajio@icici", Category.SHOPPING, ["AJIO"]),
+            ("Nykaa", "nykaa@paytm", Category.SHOPPING, ["NYKAA"]),
+            ("Meesho", "meesho@paytm", Category.SHOPPING, ["MEESHO"]),
+            ("Croma", "croma@hdfcbank", Category.SHOPPING, ["CROMA"]),
+            
+            # Grocery
+            ("Zepto", "zepto@ybl", Category.GROCERY, ["ZEPTO"]),
+            ("BigBasket", "bigbasket@ybl", Category.GROCERY, ["BIGBASKET"]),
+            ("Blinkit", "blinkit@paytm", Category.GROCERY, ["BLINKIT", "Grofers"]),
+            ("DMart", "dmart@hdfcbank", Category.GROCERY, ["DMART", "D-Mart"]),
+            ("JioMart", "jiomart@icici", Category.GROCERY, ["JIOMART"]),
+            
+            # Transport
+            ("Uber", "uber@paytm", Category.TRANSPORT, ["UBER"]),
+            ("Ola", "ola@icici", Category.TRANSPORT, ["OLA", "Ola Cabs"]),
+            ("Rapido", "rapido@ybl", Category.TRANSPORT, ["RAPIDO"]),
+            
+            # Travel
+            ("IRCTC", "irctc@sbi", Category.TRAVEL, ["IRCTC"]),
+            ("MakeMyTrip", "makemytrip@icici", Category.TRAVEL, ["MMT"]),
+            ("Goibibo", "goibibo@ybl", Category.TRAVEL, ["GOIBIBO"]),
+            ("RedBus", "redbus@paytm", Category.TRAVEL, ["REDBUS"]),
+            
+            # Fuel
+            ("IOCL", "iocl@sbi", Category.FUEL, ["Indian Oil", "IndianOil"]),
+            ("HPCL", "hpcl@hdfcbank", Category.FUEL, ["HP Petrol"]),
+            ("BPCL", "bpcl@icici", Category.FUEL, ["Bharat Petroleum"]),
+            
+            # Bills
+            ("Airtel", "airtel@paytm", Category.BILLS, ["AIRTEL"]),
+            ("Jio", "jio@icici", Category.BILLS, ["Reliance Jio"]),
+            ("Vi", "vi@ybl", Category.BILLS, ["Vodafone Idea"]),
+            ("Tata Power", "tatapower@hdfcbank", Category.BILLS, ["TATA POWER"]),
+            ("BESCOM", "bescom@ybl", Category.BILLS, ["BESCOM"]),
+            
+            # Entertainment
+            ("Netflix", "netflix@icici", Category.ENTERTAINMENT, ["NETFLIX"]),
+            ("Hotstar", "hotstar@ybl", Category.ENTERTAINMENT, ["Disney+ Hotstar"]),
+            ("Spotify", "spotify@paytm", Category.ENTERTAINMENT, ["SPOTIFY"]),
+            ("BookMyShow", "bookmyshow@paytm", Category.ENTERTAINMENT, ["BMS"]),
+            ("PVR", "pvr@hdfcbank", Category.ENTERTAINMENT, ["PVR Cinemas"]),
+            
+            # Healthcare
+            ("Apollo", "apollo@hdfcbank", Category.HEALTHCARE, ["Apollo Pharmacy"]),
+            ("PharmEasy", "pharmeasy@paytm", Category.HEALTHCARE, ["PHARMEASY"]),
+            ("1mg", "1mg@ybl", Category.HEALTHCARE, ["Tata 1mg"]),
+            
+            # Investment
+            ("Zerodha", "zerodha@hdfcbank", Category.INVESTMENT, ["ZERODHA"]),
+            ("Groww", "groww@axisbank", Category.INVESTMENT, ["GROWW"]),
+            ("Upstox", "upstox@icici", Category.INVESTMENT, ["UPSTOX"]),
+            ("Angel One", "angelone@ybl", Category.INVESTMENT, ["Angel Broking"]),
+            ("ICICI Direct", "icicidirect@icici", Category.INVESTMENT, ["ICICIdirect"]),
+            ("Paytm Money", "paytmmoney@paytm", Category.INVESTMENT, ["PAYTM MONEY"]),
+            ("5Paisa", "5paisa@icici", Category.INVESTMENT, ["5paisa"]),
+            ("Dhan", "dhan@okaxis", Category.INVESTMENT, ["DHAN"]),
+        ]
+        
+        for name, vpa, category, aliases in merchants_data:
+            merchant = Merchant(name=name, vpa=vpa, category=category, aliases=aliases)
+            self._by_category[category].append(merchant)
+            self._by_name[name.lower()] = merchant
+            self._all.append(merchant)
+    
+    def get_by_category(self, category: Category) -> List[Merchant]:
+        """Get all merchants in a category."""
+        return self._by_category.get(category, [])
+    
+    def get_random(self, rng: random.Random, category: Optional[Category] = None) -> Merchant:
+        """Get a random merchant, optionally filtered by category."""
+        if category:
+            merchants = self._by_category.get(category, self._all)
+        else:
+            merchants = self._all
+        
+        return rng.choice(merchants) if merchants else self._all[0]
+    
+    def get_by_name(self, name: str) -> Optional[Merchant]:
+        """Lookup merchant by name."""
+        return self._by_name.get(name.lower())
+
+
+class PersonDatabase:
+    """Database of Indian names for P2P transactions."""
+    
+    FIRST_NAMES = [
+        "Rahul", "Priya", "Amit", "Neha", "Vijay", "Deepak", "Anjali", "Rajesh",
+        "Sunita", "Arun", "Pooja", "Sanjay", "Kavita", "Manoj", "Rekha", "Suresh",
+        "Lakshmi", "Ganesh", "Meera", "Prakash", "Asha", "Ramesh", "Geeta", "Mohan",
+        "Savita", "Kiran", "Vinod", "Usha", "Ashok", "Padma", "Rohit", "Sneha",
+        "Vikas", "Divya", "Nitin", "Swati", "Abhishek", "Ritu", "Manish", "Preeti",
+    ]
+    
+    LAST_NAMES = [
+        "Sharma", "Singh", "Kumar", "Gupta", "Patel", "Verma", "Mehta", "Nair",
+        "Iyer", "Joshi", "Reddy", "Mishra", "Das", "Pillai", "Bose", "Menon",
+        "Venkat", "Rao", "Kulkarni", "Shah", "Patil", "Chandra", "Devi", "Lal",
+        "Sinha", "Chopra", "Saxena", "Rani", "Tiwari", "Hegde", "Agarwal", "Kapoor",
+        "Yadav", "Bansal", "Jain", "Pandey", "Malhotra", "Behera", "Sahu", "Tarai",
+    ]
+    
+    VPA_SUFFIXES = [
+        "@ybl", "@paytm", "@okicici", "@okhdfcbank", "@oksbi",
+        "@axl", "@apl", "@ibl", "@upi", "@okaxis",
+    ]
+    
+    @classmethod
+    def generate_name(cls, rng: random.Random) -> str:
+        """Generate a random Indian name."""
+        return f"{rng.choice(cls.FIRST_NAMES)} {rng.choice(cls.LAST_NAMES)}"
+    
+    @classmethod
+    def generate_vpa(cls, name: str, rng: random.Random) -> str:
+        """Generate VPA from name."""
+        parts = name.lower().split()
+        patterns = [
+            f"{parts[0]}{rng.randint(1, 99)}{rng.choice(cls.VPA_SUFFIXES)}",
+            f"{parts[0]}.{parts[-1]}{rng.choice(cls.VPA_SUFFIXES)}",
+            f"{parts[0]}{parts[-1][0]}{rng.randint(1, 9)}{rng.choice(cls.VPA_SUFFIXES)}",
+            f"{parts[0]}_{rng.randint(100, 999)}{rng.choice(cls.VPA_SUFFIXES)}",
+        ]
+        return rng.choice(patterns)
+
+
+# ============================================================================
+# BANK DATABASE
+# ============================================================================
+
+@dataclass
+class Bank:
+    """Bank entity with metadata."""
+    name: str
+    code: str
+    helpline: str
+    
+    def get_intro(self, rng: random.Random) -> str:
+        """Get bank introduction variation."""
+        intros = [
+            f"{self.name}:",
+            f"{self.name} Bank:",
+            f"{self.name} Bk:",
+            f"Alert: {self.name}",
+            f"Dear Customer, {self.name}:",
+        ]
+        return rng.choice(intros)
+
+
+class BankDatabase:
+    """Database of Indian banks."""
+    
+    BANKS = [
+        Bank("HDFC", "HDFC", "18002586161"),
+        Bank("ICICI", "ICIC", "18002662"),
+        Bank("SBI", "SBIN", "1800112211"),
+        Bank("Axis", "UTIB", "18004195959"),
+        Bank("Kotak", "KKBK", "18601266022"),
+        Bank("PNB", "PUNB", "18001802222"),
+        Bank("BOB", "BARB", "18001024455"),
+        Bank("IDFC", "IDFB", "18001024"),
+        Bank("Yes Bank", "YESB", "18001200"),
+        Bank("IndusInd", "INDB", "18602677777"),
+        Bank("Canara", "CNRB", "18004250018"),
+        Bank("Union Bank", "UBIN", "18002082244"),
+    ]
+    
+    @classmethod
+    def get_all(cls) -> List[Bank]:
+        return cls.BANKS
+    
+    @classmethod
+    def get_random(cls, rng: random.Random) -> Bank:
+        return rng.choice(cls.BANKS)
+    
+    @classmethod
+    def get_by_name(cls, name: str) -> Optional[Bank]:
+        for bank in cls.BANKS:
+            if bank.name.lower() == name.lower():
+                return bank
+        return None
+
+
+# ============================================================================
+# GROUND TRUTH SCHEMA
+# ============================================================================
+
+@dataclass
+class GroundTruth:
+    """
+    Complete ground truth for training.
+    
+    Includes all extractable fields with both raw and normalized versions.
+    """
+    # Core
+    amount: Optional[float] = None
+    amount_raw: Optional[str] = None
+    currency: str = "INR"
+    type: Optional[str] = None
+    status: str = "success"
+    
+    # Account
+    account: Optional[str] = None
+    account_raw: Optional[str] = None
+    bank: Optional[str] = None
+    
+    # DateTime
+    date: Optional[str] = None           # YYYY-MM-DD
+    date_raw: Optional[str] = None
+    time: Optional[str] = None           # HH:MM:SS
+    time_raw: Optional[str] = None
+    
+    # Reference
+    reference: Optional[str] = None
+    reference_raw: Optional[str] = None
+    reference_type: Optional[str] = None
+    
+    # Counterparty
+    merchant: Optional[str] = None
+    merchant_raw: Optional[str] = None
+    vpa: Optional[str] = None
+    beneficiary: Optional[str] = None
+    
+    # Classification
+    payment_method: Optional[str] = None
+    category: Optional[str] = None
+    
+    # Balance
+    balance_after: Optional[float] = None
+    balance_raw: Optional[str] = None
+    
+    # Metadata
+    message_type: str = "transaction"
+    is_p2m: bool = False
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert to dict, excluding None values."""
+        return {k: v for k, v in asdict(self).items() if v is not None}
+    
+    def to_json(self) -> str:
+        """Convert to JSON string."""
+        return json.dumps(self.to_dict(), ensure_ascii=False)
+
+
+# ============================================================================
+# MAIN TRANSACTION GENERATOR
+# ============================================================================
+
+class TransactionGenerator:
+    """
+    Main generator class that orchestrates all components.
+    
+    Architecture:
+    1. Sample from statistical distributions
+    2. Generate using grammar rules
+    3. Apply noise injection
+    4. Validate output
+    """
+    
+    def __init__(self, seed: int = 42):
+        self.seed = seed
+        self.rng = random.Random(seed)
+        
+        # Initialize components
+        self.grammar = MessageGrammar(seed)
+        self.noise = NoiseEngine(seed)
+        self.merchants = MerchantDatabase()
+        self.distribution = DistributionConfig()
+    
+    def generate_transaction(
+        self,
+        txn_type: Optional[TransactionType] = None,
+        status: Optional[TransactionStatus] = None,
+        category: Optional[Category] = None,
+        payment_method: Optional[PaymentMethod] = None,
+        noise_level: float = 0.3,
+    ) -> Dict[str, Any]:
+        """
+        Generate a single transaction.
+        
+        Args:
+            txn_type: Force transaction type (or sample)
+            status: Force status (or sample)
+            category: Force category (or sample)
+            payment_method: Force payment method (or sample)
+            noise_level: Amount of noise to inject
+            
+        Returns:
+            Dict with 'text' and 'ground_truth' keys
+        """
+        # Sample missing parameters from distributions
+        if txn_type is None:
+            txn_type = self.distribution.sample_weighted(
+                self.distribution.TRANSACTION_TYPE_WEIGHTS, self.rng
+            )
+        
+        if status is None:
+            status = self.distribution.sample_weighted(
+                self.distribution.STATUS_WEIGHTS, self.rng
+            )
+        
+        if category is None:
+            category = self.distribution.sample_weighted(
+                self.distribution.CATEGORY_WEIGHTS, self.rng
+            )
+        
+        if payment_method is None:
+            payment_method = self.distribution.sample_weighted(
+                self.distribution.PAYMENT_METHOD_WEIGHTS, self.rng
+            )
+        
+        # Generate components
+        bank = BankDatabase.get_random(self.rng)
+        amount = self.distribution.sample_amount(category, self.rng)
+        amount_obj = Amount.from_float(amount)
+        amount_fmt = self.rng.choice(list(AmountFormat))
+        amount_raw = amount_obj.format(amount_fmt)
+        
+        currency = self.rng.choice(list(CurrencySymbol))
+        
+        account = Account(str(self.rng.randint(1000, 9999)))
+        account_style = self.rng.choice(["XX", "xx", "****", "X"])
+        
+        # Date
+        days_ago = self.rng.randint(0, 365)
+        txn_date = date.today() - timedelta(days=days_ago)
+        time_tuple = (
+            self.rng.randint(0, 23),
+            self.rng.randint(0, 59),
+            self.rng.randint(0, 59)
+        ) if self.rng.random() > 0.5 else None
+        date_obj = TransactionDate(txn_date, time_tuple)
+        date_fmt = self.rng.choice(list(DateFormat))
+        
+        # Reference
+        ref_type = self.rng.choice(["numeric", "alphanumeric", "utr"])
+        ref = Reference.generate(ref_type, bank.code)
+        
+        # Balance
+        balance = round(self.rng.uniform(1000, 500000), 2)
+        balance_fmt = self.rng.choice([AmountFormat.COMMA_INTERNATIONAL, AmountFormat.DECIMAL_2])
+        balance_raw = Amount.from_float(balance).format(balance_fmt)
+        
+        # Counterparty
+        is_p2m = category not in [Category.TRANSFER, Category.SALARY]
+        
+        if is_p2m:
+            merchant = self.merchants.get_random(self.rng, category)
+            vpa = merchant.vpa
+            merchant_raw = merchant.get_display_name(self.rng)
+            beneficiary = None
+        else:
+            merchant = None
+            name = PersonDatabase.generate_name(self.rng)
+            vpa = PersonDatabase.generate_vpa(name, self.rng)
+            merchant_raw = None
+            beneficiary = name
+        
+        # Build parameters for grammar
+        params = {
+            'bank': bank.name,
+            'currency': currency.value,
+            'amount': amount_raw,
+            'account': account.format(account_style),
+            'date': date_obj.format_date(date_fmt),
+            'time': date_obj.format_time() or "",
+            'vpa': vpa,
+            'beneficiary': beneficiary or (merchant.name if merchant else ""),
+            'ref': ref.value,
+            'balance': balance_raw,
+            'helpline': bank.helpline,
+            'reason': self.rng.choice([
+                "Insufficient funds",
+                "Transaction declined",
+                "Network error",
+                "Invalid VPA",
+            ]) if status == TransactionStatus.FAILED else "",
+        }
+        
+        # Generate message using grammar
+        if status == TransactionStatus.FAILED:
+            text = self.grammar.generate_failed_message(params)
+        elif status == TransactionStatus.PENDING:
+            text = self.grammar.generate_pending_message(params)
+        elif txn_type == TransactionType.CREDIT:
+            text = self.grammar.generate_credit_message(params)
+        else:
+            text = self.grammar.generate_debit_message(params)
+        
+        # Apply noise
+        text = self.noise.apply(text, noise_level)
+        
+        # Build ground truth
+        ground_truth = GroundTruth(
+            amount=amount,
+            amount_raw=amount_raw,
+            currency="INR",
+            type=txn_type.value,
+            status=status.value,
+            account=account.last_four,
+            account_raw=account.format(account_style),
+            bank=bank.name,
+            date=date_obj.normalized(),
+            date_raw=date_obj.format_date(date_fmt),
+            time=f"{time_tuple[0]:02d}:{time_tuple[1]:02d}:{time_tuple[2]:02d}" if time_tuple else None,
+            time_raw=date_obj.format_time(),
+            reference=ref.value,
+            reference_type=ref_type,
+            merchant=merchant.name.lower() if merchant else None,
+            merchant_raw=merchant_raw,
+            vpa=vpa,
+            beneficiary=beneficiary,
+            payment_method=payment_method.value,
+            category=category.value,
+            balance_after=balance,
+            balance_raw=balance_raw,
+            message_type="transaction",
+            is_p2m=is_p2m,
+        )
+        
+        return {
+            'text': text,
+            'ground_truth': ground_truth.to_dict(),
+        }
+    
+    def generate_non_transaction(self, msg_type: MessageType) -> Dict[str, Any]:
+        """Generate non-transaction message (OTP, promo, etc.)."""
+        bank = BankDatabase.get_random(self.rng)
+        
+        if msg_type == MessageType.OTP:
+            otp = ''.join(self.rng.choices('0123456789', k=6))
+            templates = [
+                f"{bank.name} Bank: Your OTP is {otp}. Valid for 10 mins. Do not share with anyone.",
+                f"{bank.name}: {otp} is your OTP for transaction. Valid for 5 mins.",
+                f"OTP for {bank.name} Bank: {otp}. Do not share this with anyone.",
+            ]
+            text = self.rng.choice(templates)
+            
+        elif msg_type == MessageType.PROMOTIONAL:
+            templates = [
+                f"{bank.name} Bank wishes you a Happy Diwali! Enjoy 10% cashback on all transactions.",
+                f"Exclusive offer! Get 5% cashback on shopping using {bank.name} Credit Card.",
+                f"{bank.name}: Apply for Personal Loan at lowest interest rates. Click here.",
+            ]
+            text = self.rng.choice(templates)
+            
+        elif msg_type == MessageType.ALERT:
+            account = Account(str(self.rng.randint(1000, 9999)))
+            templates = [
+                f"{bank.name} Bank: Your Debit Card has been blocked. Call {bank.helpline}.",
+                f"{bank.name}: Suspicious activity detected on A/c {account.format('XX')}. Call immediately.",
+                f"{bank.name}: Your account KYC is pending. Update by end of month.",
+            ]
+            text = self.rng.choice(templates)
+            
+        else:  # STATEMENT
+            account = Account(str(self.rng.randint(1000, 9999)))
+            balance = round(self.rng.uniform(1000, 500000), 2)
+            text = f"{bank.name} Bank: Your A/c {account.format('XX')} balance is Rs.{balance:,.2f}."
+        
+        ground_truth = GroundTruth(
+            message_type=msg_type.value,
+            bank=bank.name,
+        )
+        
+        return {
+            'text': text,
+            'ground_truth': ground_truth.to_dict(),
+        }
+    
+    def generate_batch(
+        self,
+        count: int,
+        include_non_transactions: bool = True,
+        include_edge_cases: bool = True,
+        noise_level: float = 0.3,
+    ) -> List[Dict[str, Any]]:
+        """
+        Generate a batch of transactions.
+        
+        Args:
+            count: Number of records to generate
+            include_non_transactions: Include OTP, promo messages (5%)
+            include_edge_cases: Include edge cases (3%)
+            noise_level: Noise intensity
+        """
+        records = []
+        
+        # Calculate distribution
+        edge_case_count = int(count * 0.03) if include_edge_cases else 0
+        non_txn_count = int(count * 0.05) if include_non_transactions else 0
+        txn_count = count - edge_case_count - non_txn_count
+        
+        print(f"Generating {count:,} records...")
+        print(f"  - Transactions: {txn_count:,}")
+        print(f"  - Edge cases: {edge_case_count:,}")
+        print(f"  - Non-transactions: {non_txn_count:,}")
+        
+        # Generate transactions
+        for i in range(txn_count):
+            record = self.generate_transaction(noise_level=noise_level)
+            record['id'] = i + 1
+            record['hash'] = hashlib.sha256(record['text'].encode()).hexdigest()[:16]
+            records.append(record)
+            
+            if (i + 1) % 10000 == 0:
+                print(f"  Generated {i+1:,}/{count:,}")
+        
+        # Generate edge cases
+        edge_configs = EdgeCaseGenerator.generate_edge_case_batch(edge_case_count, self.rng)
+        for i, config in enumerate(edge_configs):
+            # Generate with edge case parameters
+            record = self.generate_transaction(noise_level=noise_level * 0.5)
+            
+            # Apply edge case modifications
+            if config.get('amount'):
+                record['ground_truth']['amount'] = config['amount']
+            if config.get('merchant'):
+                record['ground_truth']['merchant_raw'] = config['merchant']
+            if config.get('beneficiary'):
+                record['ground_truth']['beneficiary'] = config['beneficiary']
+            
+            record['id'] = txn_count + i + 1
+            record['edge_case'] = config['edge_type']
+            records.append(record)
+        
+        # Generate non-transactions
+        for i in range(non_txn_count):
+            msg_type = self.rng.choice([MessageType.OTP, MessageType.PROMOTIONAL, MessageType.ALERT])
+            record = self.generate_non_transaction(msg_type)
+            record['id'] = txn_count + edge_case_count + i + 1
+            records.append(record)
+        
+        # Shuffle
+        self.rng.shuffle(records)
+        
+        return records
+
+
+# ============================================================================
+# VALIDATION
+# ============================================================================
+
+class DatasetValidator:
+    """Validate generated dataset for quality."""
+    
+    @staticmethod
+    def validate(data: List[Dict]) -> Dict[str, Any]:
+        """
+        Validate dataset quality.
+        
+        Checks:
+        1. Required fields present
+        2. Distribution matches expectations
+        3. No duplicate texts
+        4. Format correctness
+        """
+        issues = []
+        warnings = []
+        
+        # Check duplicates
+        texts = [r['text'] for r in data]
+        unique = set(texts)
+        dup_rate = 1 - len(unique) / len(texts)
+        if dup_rate > 0.01:
+            warnings.append(f"Duplicate rate: {dup_rate:.2%}")
+        
+        # Check field completeness
+        txn_records = [r for r in data if r['ground_truth'].get('message_type') == 'transaction']
+        
+        for r in txn_records:
+            gt = r['ground_truth']
+            if gt.get('amount') is None:
+                issues.append(f"Missing amount in record {r.get('id')}")
+            if gt.get('type') is None:
+                issues.append(f"Missing type in record {r.get('id')}")
+        
+        # Check distributions
+        categories = defaultdict(int)
+        statuses = defaultdict(int)
+        types = defaultdict(int)
+        
+        for r in txn_records:
+            gt = r['ground_truth']
+            categories[gt.get('category', 'unknown')] += 1
+            statuses[gt.get('status', 'unknown')] += 1
+            types[gt.get('type', 'unknown')] += 1
+        
+        # Calculate statistics
+        total_txn = len(txn_records)
+        
+        result = {
+            'total_records': len(data),
+            'transaction_records': total_txn,
+            'non_transaction_records': len(data) - total_txn,
+            'unique_texts': len(unique),
+            'duplicate_rate': dup_rate,
+            'issues': issues[:20],
+            'warnings': warnings,
+            'distributions': {
+                'categories': {k: v/total_txn for k, v in categories.items()},
+                'statuses': {k: v/total_txn for k, v in statuses.items()},
+                'types': {k: v/total_txn for k, v in types.items()},
+            },
+            'valid': len(issues) == 0,
+        }
+        
+        return result
+
+
+# ============================================================================
+# OUTPUT WRITERS
+# ============================================================================
+
+def save_jsonl(data: List[Dict], path: Path):
+    """Save as JSONL."""
+    with open(path, 'w', encoding='utf-8') as f:
+        for record in data:
+            line = {
+                'input': record['text'],
+                'output': json.dumps(record['ground_truth'], ensure_ascii=False),
+                'id': record.get('id'),
+                'hash': record.get('hash'),
+            }
+            f.write(json.dumps(line, ensure_ascii=False) + '\n')
+    print(f"✅ Saved: {path}")
+
+
+def save_chat_format(data: List[Dict], path: Path):
+    """Save in chat format for instruction tuning."""
+    system = """Extract financial entities from Indian banking messages. Output JSON with:
+- amount, amount_raw, type, status, account, date, reference
+- merchant, vpa, beneficiary, payment_method, category
+Only include fields present in the message."""
+
+    with open(path, 'w', encoding='utf-8') as f:
+        for record in data:
+            chat = {
+                'messages': [
+                    {'role': 'system', 'content': system},
+                    {'role': 'user', 'content': record['text']},
+                    {'role': 'assistant', 'content': json.dumps(record['ground_truth'], ensure_ascii=False)},
+                ]
+            }
+            f.write(json.dumps(chat, ensure_ascii=False) + '\n')
+    print(f"✅ Saved: {path}")
+
+
+# ============================================================================
+# CLI
+# ============================================================================
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Production-grade synthetic data generator for Indian banking transactions"
+    )
+    parser.add_argument("-n", "--count", type=int, default=10000, help="Number of records")
+    parser.add_argument("-o", "--output", default="data/synthetic.jsonl", help="Output path")
+    parser.add_argument("--seed", type=int, default=42, help="Random seed")
+    parser.add_argument("--noise", type=float, default=0.3, help="Noise level (0-1)")
+    parser.add_argument("--validate", action="store_true", help="Validate after generation")
+    parser.add_argument("--chat-format", action="store_true", help="Also save chat format")
+    
+    args = parser.parse_args()
+    
+    # Create output directory
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    
+    # Generate
+    generator = TransactionGenerator(seed=args.seed)
+    data = generator.generate_batch(
+        count=args.count,
+        noise_level=args.noise,
+    )
+    
+    # Save
+    save_jsonl(data, output_path)
+    
+    if args.chat_format:
+        chat_path = output_path.with_suffix('.chat.jsonl')
+        save_chat_format(data, chat_path)
+    
+    # Validate
+    if args.validate:
+        print("\n📊 Validation:")
+        result = DatasetValidator.validate(data)
+        print(f"  Total: {result['total_records']:,}")
+        print(f"  Unique: {result['unique_texts']:,}")
+        print(f"  Duplicate rate: {result['duplicate_rate']:.2%}")
+        print(f"  Valid: {'✅' if result['valid'] else '❌'}")
+        
+        if result['issues']:
+            print(f"\n⚠️ Issues ({len(result['issues'])}):")
+            for issue in result['issues'][:5]:
+                print(f"  - {issue}")
+        
+        print("\n📈 Distributions:")
+        for dist_name, dist in result['distributions'].items():
+            print(f"\n  {dist_name}:")
+            for k, v in sorted(dist.items(), key=lambda x: -x[1])[:5]:
+                print(f"    {k}: {v:.1%}")
+
+
+if __name__ == "__main__":
+    main()

scripts/data_pipeline/step1_unify.py

@@ -0,0 +1,334 @@
+#!/usr/bin/env python3
+"""
+Step 1: Data Unification Script
+================================
+
+Reads various data formats (XML, JSON, CSV, MBOX) and
+combines them into a single standardized DataFrame.
+
+Output Schema: ['timestamp', 'sender', 'body', 'source']
+
+Usage:
+    python step1_unify.py --input /path/to/raw/data --output step1_unified.csv
+"""
+
+import argparse
+import json
+import csv
+import os
+import re
+import pandas as pd
+from pathlib import Path
+from datetime import datetime
+from typing import List, Dict, Any, Optional
+import mailbox
+import email
+from email.utils import parsedate_to_datetime
+import xml.etree.ElementTree as ET
+
+
+def parse_mbox(filepath: Path) -> List[Dict[str, Any]]:
+    """Parse Gmail MBOX export."""
+    records = []
+    try:
+        mbox = mailbox.mbox(str(filepath))
+        for message in mbox:
+            try:
+                # Get timestamp
+                date_str = message.get('Date', '')
+                try:
+                    timestamp = parsedate_to_datetime(date_str).isoformat()
+                except:
+                    timestamp = date_str
+                
+                # Get sender
+                sender = message.get('From', '')
+                
+                # Get body
+                body = ''
+                if message.is_multipart():
+                    for part in message.walk():
+                        if part.get_content_type() == 'text/plain':
+                            payload = part.get_payload(decode=True)
+                            if payload:
+                                body = payload.decode('utf-8', errors='ignore')
+                                break
+                else:
+                    payload = message.get_payload(decode=True)
+                    if payload:
+                        body = payload.decode('utf-8', errors='ignore')
+                
+                if body.strip():
+                    records.append({
+                        'timestamp': timestamp,
+                        'sender': sender,
+                        'body': body.strip(),
+                        'source': 'mbox'
+                    })
+            except Exception as e:
+                continue
+    except Exception as e:
+        print(f"  ⚠️ Error parsing MBOX {filepath}: {e}")
+    
+    return records
+
+
+def parse_json(filepath: Path) -> List[Dict[str, Any]]:
+    """Parse JSON exports (Google Takeout format)."""
+    records = []
+    try:
+        with open(filepath, 'r', encoding='utf-8') as f:
+            data = json.load(f)
+        
+        # Handle different JSON structures
+        if isinstance(data, list):
+            items = data
+        elif isinstance(data, dict):
+            # Common Google Takeout patterns
+            items = data.get('messages', []) or \
+                   data.get('transactions', []) or \
+                   data.get('items', []) or \
+                   data.get('data', []) or \
+                   [data]
+        else:
+            items = []
+        
+        for item in items:
+            if not isinstance(item, dict):
+                continue
+            
+            # Try common field names
+            timestamp = item.get('timestamp') or item.get('date') or \
+                       item.get('time') or item.get('created_at') or ''
+            
+            sender = item.get('sender') or item.get('from') or \
+                    item.get('source') or item.get('merchant') or ''
+            
+            body = item.get('body') or item.get('message') or \
+                  item.get('text') or item.get('content') or \
+                  item.get('description') or item.get('title') or ''
+            
+            # For Google Pay transactions
+            if 'amount' in item:
+                amount = item.get('amount', '')
+                merchant = item.get('merchant', {})
+                if isinstance(merchant, dict):
+                    merchant_name = merchant.get('name', '')
+                else:
+                    merchant_name = str(merchant)
+                body = f"Transaction: Rs.{amount} to {merchant_name}"
+            
+            if body and str(body).strip():
+                records.append({
+                    'timestamp': str(timestamp),
+                    'sender': str(sender),
+                    'body': str(body).strip(),
+                    'source': f'json:{filepath.name}'
+                })
+    
+    except Exception as e:
+        print(f"  ⚠️ Error parsing JSON {filepath}: {e}")
+    
+    return records
+
+
+def parse_csv(filepath: Path) -> List[Dict[str, Any]]:
+    """Parse CSV exports."""
+    records = []
+    try:
+        df = pd.read_csv(filepath, encoding='utf-8', on_bad_lines='skip')
+        
+        # Find relevant columns (case-insensitive)
+        cols = {c.lower(): c for c in df.columns}
+        
+        timestamp_col = None
+        for name in ['timestamp', 'date', 'time', 'datetime', 'created_at']:
+            if name in cols:
+                timestamp_col = cols[name]
+                break
+        
+        sender_col = None
+        for name in ['sender', 'from', 'source', 'bank', 'merchant']:
+            if name in cols:
+                sender_col = cols[name]
+                break
+        
+        body_col = None
+        for name in ['body', 'message', 'text', 'content', 'description', 'sms']:
+            if name in cols:
+                body_col = cols[name]
+                break
+        
+        if body_col:
+            for _, row in df.iterrows():
+                body = str(row.get(body_col, ''))
+                if body.strip() and body != 'nan':
+                    records.append({
+                        'timestamp': str(row.get(timestamp_col, '')) if timestamp_col else '',
+                        'sender': str(row.get(sender_col, '')) if sender_col else '',
+                        'body': body.strip(),
+                        'source': f'csv:{filepath.name}'
+                    })
+    
+    except Exception as e:
+        print(f"  ⚠️ Error parsing CSV {filepath}: {e}")
+    
+    return records
+
+
+def parse_xml(filepath: Path) -> List[Dict[str, Any]]:
+    """Parse XML exports (SMS Backup format)."""
+    records = []
+    try:
+        tree = ET.parse(filepath)
+        root = tree.getroot()
+        
+        # Common SMS backup format
+        for sms in root.findall('.//sms') or root.findall('.//message'):
+            body = sms.get('body') or sms.text or ''
+            timestamp = sms.get('date') or sms.get('timestamp') or ''
+            sender = sms.get('address') or sms.get('sender') or sms.get('from') or ''
+            
+            if body.strip():
+                # Convert timestamp if it's milliseconds
+                if timestamp.isdigit() and len(timestamp) > 10:
+                    try:
+                        timestamp = datetime.fromtimestamp(int(timestamp)/1000).isoformat()
+                    except:
+                        pass
+                
+                records.append({
+                    'timestamp': timestamp,
+                    'sender': sender,
+                    'body': body.strip(),
+                    'source': f'xml:{filepath.name}'
+                })
+    
+    except Exception as e:
+        print(f"  ⚠️ Error parsing XML {filepath}: {e}")
+    
+    return records
+
+
+def find_all_files(input_dir: Path) -> Dict[str, List[Path]]:
+    """Find all data files recursively."""
+    files = {
+        'mbox': [],
+        'json': [],
+        'csv': [],
+        'xml': []
+    }
+    
+    for filepath in input_dir.rglob('*'):
+        if filepath.is_file():
+            ext = filepath.suffix.lower()
+            if ext == '.mbox':
+                files['mbox'].append(filepath)
+            elif ext == '.json':
+                files['json'].append(filepath)
+            elif ext == '.csv':
+                files['csv'].append(filepath)
+            elif ext == '.xml':
+                files['xml'].append(filepath)
+    
+    return files
+
+
+def unify_data(input_dir: Path) -> pd.DataFrame:
+    """Main function to unify all data sources."""
+    print("=" * 60)
+    print("📂 STEP 1: DATA UNIFICATION")
+    print("=" * 60)
+    
+    all_records = []
+    
+    # Find all files
+    print(f"\n🔍 Scanning: {input_dir}")
+    files = find_all_files(input_dir)
+    
+    total_files = sum(len(v) for v in files.values())
+    print(f"   Found {total_files} files to process")
+    
+    # Parse MBOX files
+    if files['mbox']:
+        print(f"\n📧 Processing {len(files['mbox'])} MBOX files...")
+        for f in files['mbox']:
+            print(f"   Processing: {f.name}")
+            records = parse_mbox(f)
+            all_records.extend(records)
+            print(f"   ✅ Extracted {len(records)} messages")
+    
+    # Parse JSON files
+    if files['json']:
+        print(f"\n📋 Processing {len(files['json'])} JSON files...")
+        for f in files['json']:
+            print(f"   Processing: {f.name}")
+            records = parse_json(f)
+            all_records.extend(records)
+            print(f"   ✅ Extracted {len(records)} records")
+    
+    # Parse CSV files
+    if files['csv']:
+        print(f"\n📊 Processing {len(files['csv'])} CSV files...")
+        for f in files['csv']:
+            print(f"   Processing: {f.name}")
+            records = parse_csv(f)
+            all_records.extend(records)
+            print(f"   ✅ Extracted {len(records)} records")
+    
+    # Parse XML files
+    if files['xml']:
+        print(f"\n📝 Processing {len(files['xml'])} XML files...")
+        for f in files['xml']:
+            print(f"   Processing: {f.name}")
+            records = parse_xml(f)
+            all_records.extend(records)
+            print(f"   ✅ Extracted {len(records)} records")
+    
+    # Create DataFrame
+    df = pd.DataFrame(all_records, columns=['timestamp', 'sender', 'body', 'source'])
+    
+    # Remove exact duplicates
+    original_count = len(df)
+    df = df.drop_duplicates(subset=['body'])
+    dedup_count = len(df)
+    
+    print(f"\n📊 SUMMARY:")
+    print(f"   Total records: {original_count}")
+    print(f"   After dedup:   {dedup_count}")
+    print(f"   Removed:       {original_count - dedup_count} duplicates")
+    
+    return df
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Step 1: Unify data sources")
+    parser.add_argument("--input", "-i", required=True, help="Input directory with raw data")
+    parser.add_argument("--output", "-o", default="data/pipeline/step1_unified.csv", 
+                       help="Output CSV path")
+    args = parser.parse_args()
+    
+    input_dir = Path(args.input)
+    if not input_dir.exists():
+        print(f"❌ Input directory not found: {input_dir}")
+        return
+    
+    # Unify data
+    df = unify_data(input_dir)
+    
+    if len(df) == 0:
+        print("\n❌ No data extracted! Check your input directory.")
+        return
+    
+    # Save output
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    df.to_csv(output_path, index=False)
+    
+    print(f"\n✅ Saved to: {output_path}")
+    print(f"   Records: {len(df)}")
+    print("\nNext: python scripts/data_pipeline/step2_filter.py")
+
+
+if __name__ == "__main__":
+    main()

scripts/data_pipeline/step2_filter.py

@@ -0,0 +1,295 @@
+#!/usr/bin/env python3
+"""
+Step 2: Garbage Filter
+======================
+
+Filters out noise from unified data:
+- OTPs / Verification codes
+- Login alerts
+- Marketing spam
+- Bill reminders (non-transaction)
+- Password reset
+
+Keeps only actual transaction messages.
+
+Usage:
+    python step2_filter.py --input step1_unified.csv --output step2_training_ready.csv
+"""
+
+import argparse
+import re
+import pandas as pd
+from pathlib import Path
+from typing import Tuple
+
+
+# ============================================================================
+# GARBAGE PATTERNS (Messages to REMOVE)
+# ============================================================================
+
+GARBAGE_PATTERNS = [
+    # OTPs and verification codes
+    r'\bOTP\b',
+    r'\bone.time.password\b',
+    r'\bverification.code\b',
+    r'\bverify.your\b',
+    r'\bconfirm.your.identity\b',
+    r'\bcvv\b',
+    r'\bpin\b.*\bdo.not.share\b',
+    r'\b\d{4,6}\b.*\bexpires?\b',
+    r'\bvalid.for.\d+.min',
+    
+    # Login/Security alerts
+    r'\blogin.alert\b',
+    r'\blogged.in\b',
+    r'\bnew.device\b',
+    r'\bnew.login\b',
+    r'\bsecurity.alert\b',
+    r'\bsign.in.attempt\b',
+    r'\bpassword.changed\b',
+    r'\bpassword.reset\b',
+    r'\baccount.locked\b',
+    r'\bunusual.activity\b',
+    
+    # Marketing / Promotional
+    r'\b\d+%.off\b',
+    r'\bdiscount\b',
+    r'\bcashback.offer\b',
+    r'\bflat.rs\b.*\boff\b',
+    r'\bget.upto\b',
+    r'\bwin.upto\b',
+    r'\bexclusive.offer\b',
+    r'\blimited.time\b',
+    r'\bspecial.offer\b',
+    r'\bsale.live\b',
+    r'\bshop.now\b',
+    r'\bbuy.now\b',
+    r'\bsubscribe\b',
+    r'\bunsubscribe\b',
+    r'\bnewsletter\b',
+    
+    # Bill reminders (not actual transactions)
+    r'\bbill.due\b',
+    r'\bbill.reminder\b',
+    r'\bdue.date\b',
+    r'\bpay.before\b',
+    r'\bauto.debit.scheduled\b',
+    r'\bemi.due\b',
+    r'\bpayment.reminder\b',
+    r'\bminimum.amount.due\b',
+    r'\boutstanding.balance\b',
+    
+    # Account statements / Summaries
+    r'\baccount.statement\b',
+    r'\bmonthly.statement\b',
+    r'\be.statement\b',
+    r'\bstatement.ready\b',
+    r'\bdownload.statement\b',
+    
+    # Delivery / Shipping (Not finance)
+    r'\bout.for.delivery\b',
+    r'\bdelivered.to\b',
+    r'\bshipment.update\b',
+    r'\btracking.number\b',
+    r'\border.confirmed\b',
+    r'\border.placed\b',
+    r'\border.shipped\b',
+    
+    # App notifications
+    r'\brate.your.experience\b',
+    r'\bleave.a.review\b',
+    r'\bupdate.available\b',
+    r'\bapp.update\b',
+    r'\bdownload.app\b',
+    r'\binstall.app\b',
+    
+    # Generic noise
+    r'\bclick.here\b',
+    r'\bvisit.us\b',
+    r'\bcall.us\b',
+    r'\bcontact.us\b',
+    r'\bfollow.us\b',
+    r'\bjoin.us\b',
+    r'\blearn.more\b',
+    r'\bread.more\b',
+]
+
+# ============================================================================
+# KEEP PATTERNS (Messages to KEEP - actual transactions)
+# ============================================================================
+
+KEEP_PATTERNS = [
+    # Transaction keywords
+    r'\bdebited?\b',
+    r'\bcredited?\b',
+    r'\btransferred?\b',
+    r'\bwithdra(?:wn|wal)\b',
+    r'\bdeposited?\b',
+    r'\breceived?\b',
+    r'\bsent\b',
+    r'\bpaid\b',
+    r'\bpurchase\b',
+    r'\btransaction\b',
+    r'\btxn\b',
+    r'\bupi\b',
+    r'\bneft\b',
+    r'\bimps\b',
+    r'\brtgs\b',
+    
+    # Amount patterns
+    r'rs\.?\s*[\d,]+',
+    r'inr\s*[\d,]+',
+    r'₹\s*[\d,]+',
+    
+    # Account references
+    r'a/c\s*\w+',
+    r'acct?\s*\w+',
+    r'account\s*\w+',
+    
+    # Reference numbers
+    r'ref\.?\s*:?\s*\d{10,}',
+    r'upi.?ref\s*:?\s*\d{10,}',
+]
+
+
+def is_garbage(text: str) -> bool:
+    """Check if message is garbage (should be removed)."""
+    text_lower = text.lower()
+    
+    for pattern in GARBAGE_PATTERNS:
+        if re.search(pattern, text_lower):
+            return True
+    
+    return False
+
+
+def is_transaction(text: str) -> bool:
+    """Check if message is a transaction (should be kept)."""
+    text_lower = text.lower()
+    
+    matches = 0
+    for pattern in KEEP_PATTERNS:
+        if re.search(pattern, text_lower):
+            matches += 1
+    
+    # Require at least 2 transaction indicators
+    return matches >= 2
+
+
+def classify_message(text: str) -> Tuple[str, str]:
+    """
+    Classify a message as 'keep', 'garbage', or 'uncertain'.
+    
+    Returns: (classification, reason)
+    """
+    if not text or len(text.strip()) < 20:
+        return 'garbage', 'too_short'
+    
+    # First check garbage patterns (high confidence negative)
+    if is_garbage(text):
+        return 'garbage', 'matched_garbage_pattern'
+    
+    # Then check transaction patterns (high confidence positive)
+    if is_transaction(text):
+        return 'keep', 'matched_transaction_pattern'
+    
+    # Messages with amounts but no clear transaction type
+    if re.search(r'(?:rs\.?|inr|₹)\s*[\d,]+', text.lower()):
+        return 'uncertain', 'has_amount_no_transaction_type'
+    
+    # Default: uncertain
+    return 'uncertain', 'no_strong_signals'
+
+
+def filter_data(df: pd.DataFrame) -> Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:
+    """
+    Filter DataFrame into keep/garbage/uncertain.
+    
+    Returns: (keep_df, garbage_df, uncertain_df)
+    """
+    print("=" * 60)
+    print("🧹 STEP 2: GARBAGE FILTER")
+    print("=" * 60)
+    
+    results = []
+    for _, row in df.iterrows():
+        body = row.get('body', '')
+        classification, reason = classify_message(str(body))
+        results.append({
+            **row.to_dict(),
+            'classification': classification,
+            'filter_reason': reason
+        })
+    
+    result_df = pd.DataFrame(results)
+    
+    keep_df = result_df[result_df['classification'] == 'keep'].copy()
+    garbage_df = result_df[result_df['classification'] == 'garbage'].copy()
+    uncertain_df = result_df[result_df['classification'] == 'uncertain'].copy()
+    
+    # Print summary
+    total = len(df)
+    print(f"\n📊 FILTER RESULTS:")
+    print(f"   Total input:  {total:,}")
+    print(f"   ✅ Keep:      {len(keep_df):,} ({100*len(keep_df)/total:.1f}%)")
+    print(f"   ❌ Garbage:   {len(garbage_df):,} ({100*len(garbage_df)/total:.1f}%)")
+    print(f"   ❓ Uncertain: {len(uncertain_df):,} ({100*len(uncertain_df)/total:.1f}%)")
+    
+    # Show reason breakdown for garbage
+    if len(garbage_df) > 0:
+        print(f"\n📋 Garbage Reasons:")
+        for reason, count in garbage_df['filter_reason'].value_counts().items():
+            print(f"   {reason}: {count:,}")
+    
+    return keep_df, garbage_df, uncertain_df
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Step 2: Filter garbage messages")
+    parser.add_argument("--input", "-i", default="data/pipeline/step1_unified.csv",
+                       help="Input CSV from step 1")
+    parser.add_argument("--output", "-o", default="data/pipeline/step2_training_ready.csv",
+                       help="Output CSV with clean data")
+    parser.add_argument("--save-all", action="store_true",
+                       help="Also save garbage and uncertain to separate files")
+    args = parser.parse_args()
+    
+    input_path = Path(args.input)
+    if not input_path.exists():
+        print(f"❌ Input file not found: {input_path}")
+        print(f"   Run step1_unify.py first!")
+        return
+    
+    # Load data
+    print(f"\n📂 Loading: {input_path}")
+    df = pd.read_csv(input_path)
+    print(f"   Loaded {len(df):,} records")
+    
+    # Filter
+    keep_df, garbage_df, uncertain_df = filter_data(df)
+    
+    # Save results
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    
+    # Drop helper columns before saving
+    keep_df = keep_df.drop(columns=['classification', 'filter_reason'])
+    keep_df.to_csv(output_path, index=False)
+    print(f"\n✅ Saved clean data to: {output_path}")
+    print(f"   Records: {len(keep_df):,}")
+    
+    if args.save_all:
+        garbage_path = output_path.parent / "step2_garbage.csv"
+        uncertain_path = output_path.parent / "step2_uncertain.csv"
+        
+        garbage_df.to_csv(garbage_path, index=False)
+        uncertain_df.to_csv(uncertain_path, index=False)
+        print(f"\n📁 Also saved:")
+        print(f"   Garbage: {garbage_path}")
+        print(f"   Uncertain: {uncertain_path}")
+    
+    print("\nNext: python scripts/data_pipeline/step3_baseline.py")
+
+
+if __name__ == "__main__":
+    main()

scripts/data_pipeline/step3_baseline.py

@@ -0,0 +1,260 @@
+#!/usr/bin/env python3
+"""
+Step 3: Baseline Test
+=====================
+
+Tests the current finee extractor on the cleaned training data
+to establish a baseline before fine-tuning.
+
+This answers: "How good is our Regex engine on real data?"
+
+Usage:
+    python step3_baseline.py --input step2_training_ready.csv
+"""
+
+import argparse
+import json
+import pandas as pd
+from pathlib import Path
+from typing import Dict, Any, List
+from datetime import datetime
+import sys
+
+# Add parent to path for imports
+sys.path.insert(0, str(Path(__file__).parent.parent.parent))
+
+try:
+    from finee import extract
+    from finee.schema import ExtractionResult
+except ImportError:
+    print("❌ finee not installed!")
+    print("   Run: pip install finee")
+    sys.exit(1)
+
+
+def extract_and_analyze(row: Dict[str, Any]) -> Dict[str, Any]:
+    """Extract entities from a message and analyze results."""
+    body = str(row.get('body', ''))
+    
+    try:
+        result = extract(body)
+        
+        return {
+            # Original data
+            'timestamp': row.get('timestamp', ''),
+            'sender': row.get('sender', ''),
+            'body': body[:200] + '...' if len(body) > 200 else body,
+            'source': row.get('source', ''),
+            
+            # Extracted fields
+            'extracted_amount': result.amount,
+            'extracted_type': result.type.value if result.type else None,
+            'extracted_account': result.account,
+            'extracted_date': result.date,
+            'extracted_reference': result.reference,
+            'extracted_vpa': result.vpa,
+            'extracted_merchant': result.merchant,
+            'extracted_category': result.category.value if result.category else None,
+            'extracted_confidence': result.confidence.value if result.confidence else None,
+            'extracted_confidence_score': result.confidence_score,
+            
+            # Quality metrics
+            'has_amount': result.amount is not None,
+            'has_type': result.type is not None,
+            'has_merchant': result.merchant is not None,
+            'has_category': result.category is not None,
+            'fields_extracted': sum([
+                result.amount is not None,
+                result.type is not None,
+                result.account is not None,
+                result.date is not None,
+                result.reference is not None,
+                result.merchant is not None,
+                result.category is not None,
+            ]),
+            
+            # Processing info
+            'processing_time_ms': result.processing_time_ms,
+            'extraction_success': result.amount is not None and result.type is not None,
+        }
+    except Exception as e:
+        return {
+            'timestamp': row.get('timestamp', ''),
+            'sender': row.get('sender', ''),
+            'body': body[:200],
+            'source': row.get('source', ''),
+            'extraction_error': str(e),
+            'extraction_success': False,
+        }
+
+
+def run_baseline(df: pd.DataFrame) -> pd.DataFrame:
+    """Run baseline extraction on all rows."""
+    print("=" * 60)
+    print("📊 STEP 3: BASELINE TEST")
+    print("=" * 60)
+    print(f"\nTesting finee extractor on {len(df):,} messages...")
+    print("(This tests Regex-only mode, no LLM)\n")
+    
+    results = []
+    success_count = 0
+    
+    for i, (_, row) in enumerate(df.iterrows()):
+        result = extract_and_analyze(row.to_dict())
+        results.append(result)
+        
+        if result.get('extraction_success'):
+            success_count += 1
+        
+        # Progress every 100
+        if (i + 1) % 100 == 0:
+            pct = 100 * success_count / (i + 1)
+            print(f"   Processed {i+1:,}/{len(df):,} ({pct:.1f}% success rate)")
+    
+    return pd.DataFrame(results)
+
+
+def analyze_results(results_df: pd.DataFrame) -> Dict[str, Any]:
+    """Analyze extraction results."""
+    total = len(results_df)
+    
+    # Core metrics
+    success_count = results_df['extraction_success'].sum()
+    has_amount = results_df['has_amount'].sum()
+    has_type = results_df['has_type'].sum()
+    has_merchant = results_df['has_merchant'].sum()
+    has_category = results_df['has_category'].sum()
+    
+    # Confidence distribution
+    confidence_counts = results_df['extracted_confidence'].value_counts().to_dict()
+    
+    # Type distribution
+    type_counts = results_df['extracted_type'].value_counts().to_dict()
+    
+    # Category distribution
+    category_counts = results_df['extracted_category'].value_counts().to_dict()
+    
+    # Top merchants
+    merchant_counts = results_df['extracted_merchant'].value_counts().head(20).to_dict()
+    
+    # Performance
+    avg_time = results_df['processing_time_ms'].mean()
+    
+    analysis = {
+        'total_messages': total,
+        'extraction_success_rate': 100 * success_count / total,
+        'field_coverage': {
+            'amount': 100 * has_amount / total,
+            'type': 100 * has_type / total,
+            'merchant': 100 * has_merchant / total,
+            'category': 100 * has_category / total,
+        },
+        'confidence_distribution': confidence_counts,
+        'type_distribution': type_counts,
+        'category_distribution': category_counts,
+        'top_merchants': merchant_counts,
+        'avg_processing_time_ms': avg_time,
+        'timestamp': datetime.now().isoformat(),
+    }
+    
+    return analysis
+
+
+def print_analysis(analysis: Dict[str, Any]) -> None:
+    """Print analysis results."""
+    print("\n" + "=" * 60)
+    print("📈 BASELINE RESULTS")
+    print("=" * 60)
+    
+    print(f"\n📊 COVERAGE:")
+    print(f"   Total messages:      {analysis['total_messages']:,}")
+    print(f"   Extraction success:  {analysis['extraction_success_rate']:.1f}%")
+    
+    print(f"\n📋 FIELD COVERAGE:")
+    for field, pct in analysis['field_coverage'].items():
+        status = "✅" if pct >= 80 else "⚠️" if pct >= 50 else "❌"
+        print(f"   {field:12} {pct:5.1f}% {status}")
+    
+    print(f"\n📊 CONFIDENCE DISTRIBUTION:")
+    for level, count in sorted(analysis['confidence_distribution'].items(), key=lambda x: -x[1]):
+        if level:
+            pct = 100 * count / analysis['total_messages']
+            print(f"   {level:10} {count:,} ({pct:.1f}%)")
+    
+    print(f"\n💳 TRANSACTION TYPES:")
+    for txn_type, count in sorted(analysis['type_distribution'].items(), key=lambda x: -x[1]):
+        if txn_type:
+            pct = 100 * count / analysis['total_messages']
+            print(f"   {txn_type:10} {count:,} ({pct:.1f}%)")
+    
+    print(f"\n🏪 TOP 10 MERCHANTS:")
+    for i, (merchant, count) in enumerate(list(analysis['top_merchants'].items())[:10]):
+        if merchant:
+            print(f"   {i+1:2}. {merchant:20} {count:,}")
+    
+    print(f"\n⚡ PERFORMANCE:")
+    print(f"   Avg processing time: {analysis['avg_processing_time_ms']:.2f}ms")
+    print(f"   Throughput: ~{1000/analysis['avg_processing_time_ms']:.0f} msg/sec")
+    
+    print("\n" + "=" * 60)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Step 3: Baseline extraction test")
+    parser.add_argument("--input", "-i", default="data/pipeline/step2_training_ready.csv",
+                       help="Input CSV from step 2")
+    parser.add_argument("--output", "-o", default="data/pipeline/step3_baseline_results.csv",
+                       help="Output CSV with extraction results")
+    parser.add_argument("--limit", "-n", type=int, default=None,
+                       help="Limit number of rows to process (for testing)")
+    args = parser.parse_args()
+    
+    input_path = Path(args.input)
+    if not input_path.exists():
+        print(f"❌ Input file not found: {input_path}")
+        print(f"   Run step2_filter.py first!")
+        return
+    
+    # Load data
+    print(f"\n📂 Loading: {input_path}")
+    df = pd.read_csv(input_path)
+    
+    if args.limit:
+        df = df.head(args.limit)
+        print(f"   (Limited to {args.limit} rows for testing)")
+    
+    print(f"   Loaded {len(df):,} records")
+    
+    # Run baseline
+    results_df = run_baseline(df)
+    
+    # Analyze
+    analysis = analyze_results(results_df)
+    print_analysis(analysis)
+    
+    # Save results
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    results_df.to_csv(output_path, index=False)
+    print(f"\n✅ Saved extraction results to: {output_path}")
+    
+    # Save analysis as JSON
+    analysis_path = output_path.parent / "step3_baseline_analysis.json"
+    with open(analysis_path, 'w') as f:
+        json.dump(analysis, f, indent=2, default=str)
+    print(f"   Analysis saved to: {analysis_path}")
+    
+    # Summary
+    success_rate = analysis['extraction_success_rate']
+    if success_rate >= 80:
+        print(f"\n🎉 Great! {success_rate:.1f}% success rate. Regex is working well!")
+    elif success_rate >= 50:
+        print(f"\n⚠️ {success_rate:.1f}% success rate. Room for improvement.")
+        print("   Consider adding more regex patterns or enabling LLM mode.")
+    else:
+        print(f"\n❌ Low {success_rate:.1f}% success rate.")
+        print("   Your data may have unusual formats. Review failed extractions.")
+
+
+if __name__ == "__main__":
+    main()

scripts/data_pipeline/step4_label.py

@@ -0,0 +1,302 @@
+#!/usr/bin/env python3
+"""
+Step 4: Create Labeled Training Data
+=====================================
+
+Processes the clean SMS data and creates training labels.
+Extracts:
+- amount, type, account, date, reference (regex)
+- beneficiary_name (from SMS pattern)
+- Detects merchant vs P2P transactions
+
+Usage:
+    python step4_label.py --input step2_sms_clean.csv --output step4_labeled.csv
+"""
+
+import argparse
+import re
+import json
+import pandas as pd
+from pathlib import Path
+from typing import Dict, Any, Optional, Tuple
+
+
+# ============================================================================
+# ICICI BANK SMS PATTERNS (Dominant format in data)
+# ============================================================================
+
+ICICI_DEBIT_PATTERN = re.compile(
+    r'ICICI Bank Acc?t?\s*XX?(\d+)\s+debited\s+(?:for\s+)?Rs\.?\s*([\d,]+(?:\.\d{2})?)\s+on\s+(\d{1,2}-[A-Za-z]{3}-\d{2,4})[\s;]+([A-Za-z0-9\s]+?)\s+credited\.\s*UPI[:\s]*(\d+)',
+    re.IGNORECASE
+)
+
+ICICI_CREDIT_PATTERN = re.compile(
+    r'(?:Dear Customer,?\s*)?Acc?t?\s*XX?(\d+)\s+(?:is\s+)?credited\s+(?:with\s+)?Rs\.?\s*([\d,]+(?:\.\d{2})?)\s+on\s+(\d{1,2}-[A-Za-z]{3}-\d{2,4})\s+from\s+([A-Za-z0-9\s]+?)[\.\s]+UPI[:\s]*(\d+)',
+    re.IGNORECASE
+)
+
+# Generic amount pattern
+AMOUNT_PATTERN = re.compile(r'Rs\.?\s*([\d,]+(?:\.\d{1,2})?)', re.IGNORECASE)
+DATE_PATTERN = re.compile(r'(\d{1,2}[-/][A-Za-z]{3}[-/]\d{2,4}|\d{1,2}[-/]\d{1,2}[-/]\d{2,4})')
+UPI_REF_PATTERN = re.compile(r'UPI[:\s]*(\d{12,16})', re.IGNORECASE)
+ACCOUNT_PATTERN = re.compile(r'XX?(\d{3,4})', re.IGNORECASE)
+
+# Merchant detection keywords (P2M vs P2P)
+MERCHANT_KEYWORDS = {
+    'swiggy', 'zomato', 'uber', 'ola', 'amazon', 'flipkart', 'paytm', 
+    'phonepe', 'google', 'youtube', 'netflix', 'spotify', 'airtel',
+    'jio', 'vodafone', 'bsnl', 'electricity', 'gas', 'water', 'bill',
+    'store', 'mart', 'shop', 'restaurant', 'hotel', 'hospital', 'clinic',
+    'pharmacy', 'petrol', 'fuel', 'charging', 'parking', 'toll', 'metro',
+    'railway', 'flight', 'bus', 'cab', 'taxi', 'rent', 'insurance',
+    'zepto', 'bigbasket', 'blinkit', 'instamart', 'dunzo', 'myntra',
+    'ajio', 'nykaa', 'tata', 'reliance', 'dmart', 'more', 'grofers'
+}
+
+
+def is_merchant(beneficiary: str) -> bool:
+    """Determine if beneficiary is a merchant (P2M) or person (P2P)."""
+    if not beneficiary:
+        return False
+    
+    name_lower = beneficiary.lower().strip()
+    
+    # Check against known merchant keywords
+    for keyword in MERCHANT_KEYWORDS:
+        if keyword in name_lower:
+            return True
+    
+    # Heuristics for P2M vs P2P:
+    # - Merchants often have Ltd, Pvt, Inc, Store, Shop
+    # - Person names are usually 2-3 words
+    
+    if any(x in name_lower for x in ['ltd', 'pvt', 'inc', 'llp', 'corp', 
+                                      'store', 'shop', 'mart', 'services',
+                                      'limited', 'private']):
+        return True
+    
+    # All caps names with numbers are likely merchants
+    if beneficiary.isupper() and any(c.isdigit() for c in beneficiary):
+        return True
+    
+    return False
+
+
+def normalize_beneficiary(name: str) -> str:
+    """Clean up beneficiary name."""
+    if not name:
+        return ""
+    
+    # Remove trailing/leading whitespace
+    name = name.strip()
+    
+    # Remove common suffixes
+    name = re.sub(r'\s+credited\.?$', '', name, flags=re.IGNORECASE)
+    name = re.sub(r'\s+debited\.?$', '', name, flags=re.IGNORECASE)
+    
+    # Title case if all uppercase
+    if name.isupper():
+        name = name.title()
+    
+    return name.strip()
+
+
+def extract_from_sms(body: str) -> Dict[str, Any]:
+    """Extract all fields from SMS body."""
+    result = {
+        'amount': None,
+        'type': None,
+        'account': None,
+        'date': None,
+        'reference': None,
+        'beneficiary': None,
+        'is_merchant': False,
+        'category': None,
+        'extraction_method': None
+    }
+    
+    # Try ICICI debit pattern
+    match = ICICI_DEBIT_PATTERN.search(body)
+    if match:
+        result['account'] = match.group(1)
+        result['amount'] = float(match.group(2).replace(',', ''))
+        result['date'] = match.group(3)
+        result['beneficiary'] = normalize_beneficiary(match.group(4))
+        result['reference'] = match.group(5)
+        result['type'] = 'debit'
+        result['is_merchant'] = is_merchant(result['beneficiary'])
+        result['extraction_method'] = 'icici_debit_pattern'
+        return result
+    
+    # Try ICICI credit pattern
+    match = ICICI_CREDIT_PATTERN.search(body)
+    if match:
+        result['account'] = match.group(1)
+        result['amount'] = float(match.group(2).replace(',', ''))
+        result['date'] = match.group(3)
+        result['beneficiary'] = normalize_beneficiary(match.group(4))
+        result['reference'] = match.group(5)
+        result['type'] = 'credit'
+        result['is_merchant'] = is_merchant(result['beneficiary'])
+        result['extraction_method'] = 'icici_credit_pattern'
+        return result
+    
+    # Fallback: generic extraction
+    # Amount
+    amount_match = AMOUNT_PATTERN.search(body)
+    if amount_match:
+        try:
+            result['amount'] = float(amount_match.group(1).replace(',', ''))
+        except:
+            pass
+    
+    # Type
+    if re.search(r'\bdebit', body, re.IGNORECASE):
+        result['type'] = 'debit'
+    elif re.search(r'\bcredit', body, re.IGNORECASE):
+        result['type'] = 'credit'
+    
+    # Account
+    acc_match = ACCOUNT_PATTERN.search(body)
+    if acc_match:
+        result['account'] = acc_match.group(1)
+    
+    # Date
+    date_match = DATE_PATTERN.search(body)
+    if date_match:
+        result['date'] = date_match.group(1)
+    
+    # Reference
+    ref_match = UPI_REF_PATTERN.search(body)
+    if ref_match:
+        result['reference'] = ref_match.group(1)
+    
+    result['extraction_method'] = 'generic_fallback'
+    return result
+
+
+def create_training_label(row: Dict[str, Any], extraction: Dict[str, Any]) -> Dict[str, Any]:
+    """Create a training label with ground truth."""
+    body = str(row.get('body', ''))
+    
+    # Build ground truth JSON (what we want the model to output)
+    ground_truth = {
+        'amount': extraction['amount'],
+        'type': extraction['type'],
+        'account': extraction['account'],
+        'date': extraction['date'],
+        'reference': extraction['reference'],
+        'beneficiary': extraction['beneficiary'],
+        'is_p2m': extraction['is_merchant'],
+    }
+    
+    # Remove None values
+    ground_truth = {k: v for k, v in ground_truth.items() if v is not None}
+    
+    return {
+        # Original data
+        'timestamp': row.get('timestamp', ''),
+        'sender': row.get('sender', ''),
+        'body': body,
+        'source': row.get('source', ''),
+        
+        # Extracted fields
+        **{f'extracted_{k}': v for k, v in extraction.items()},
+        
+        # Training label (JSON format for LLM fine-tuning)
+        'ground_truth_json': json.dumps(ground_truth, ensure_ascii=False),
+        
+        # Quality flags
+        'has_amount': extraction['amount'] is not None,
+        'has_type': extraction['type'] is not None,
+        'has_beneficiary': extraction['beneficiary'] is not None and len(extraction['beneficiary']) > 0,
+        'complete_extraction': all([
+            extraction['amount'] is not None,
+            extraction['type'] is not None,
+            extraction['reference'] is not None
+        ]),
+    }
+
+
+def label_data(df: pd.DataFrame) -> pd.DataFrame:
+    """Label all data for training."""
+    print("=" * 60)
+    print("🏷️ STEP 4: CREATING LABELED TRAINING DATA")
+    print("=" * 60)
+    
+    results = []
+    complete_count = 0
+    
+    for i, (_, row) in enumerate(df.iterrows()):
+        body = str(row.get('body', ''))
+        extraction = extract_from_sms(body)
+        label = create_training_label(row.to_dict(), extraction)
+        results.append(label)
+        
+        if label['complete_extraction']:
+            complete_count += 1
+        
+        if (i + 1) % 500 == 0:
+            print(f"   Processed {i+1:,}/{len(df):,} ({100*complete_count/(i+1):.1f}% complete)")
+    
+    result_df = pd.DataFrame(results)
+    
+    print(f"\n📊 LABELING RESULTS:")
+    print(f"   Total records: {len(result_df):,}")
+    print(f"   Complete extractions: {complete_count:,} ({100*complete_count/len(result_df):.1f}%)")
+    print(f"   Has amount: {result_df['has_amount'].sum():,}")
+    print(f"   Has type: {result_df['has_type'].sum():,}")
+    print(f"   Has beneficiary: {result_df['has_beneficiary'].sum():,}")
+    
+    # Show breakdown by extraction method
+    print(f"\n📋 EXTRACTION METHODS:")
+    method_counts = result_df['extracted_extraction_method'].value_counts()
+    for method, count in method_counts.items():
+        print(f"   {method}: {count:,}")
+    
+    return result_df
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Step 4: Create labeled training data")
+    parser.add_argument("--input", "-i", default="data/pipeline/step2_sms_clean.csv",
+                       help="Input CSV from step 2 (SMS only)")
+    parser.add_argument("--output", "-o", default="data/pipeline/step4_labeled.csv",
+                       help="Output CSV with labels")
+    args = parser.parse_args()
+    
+    input_path = Path(args.input)
+    if not input_path.exists():
+        print(f"❌ Input file not found: {input_path}")
+        return
+    
+    # Load data
+    print(f"\n📂 Loading: {input_path}")
+    df = pd.read_csv(input_path)
+    print(f"   Loaded {len(df):,} records")
+    
+    # Label data
+    labeled_df = label_data(df)
+    
+    # Save output
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    labeled_df.to_csv(output_path, index=False)
+    
+    print(f"\n✅ Saved labeled data to: {output_path}")
+    
+    # Also save training-ready JSONL (for LLM fine-tuning)
+    jsonl_path = output_path.parent / "step4_training.jsonl"
+    with open(jsonl_path, 'w') as f:
+        for _, row in labeled_df[labeled_df['complete_extraction']].iterrows():
+            training_example = {
+                'input': row['body'],
+                'output': row['ground_truth_json']
+            }
+            f.write(json.dumps(training_example, ensure_ascii=False) + '\n')
+    
+    print(f"   JSONL for LLM training: {jsonl_path}")
+
+
+if __name__ == "__main__":
+    main()

scripts/finetune.py

@@ -0,0 +1,578 @@
+#!/usr/bin/env python3
+"""
+LLM Fine-tuning Pipeline for Finance Entity Extraction
+=======================================================
+
+Fine-tunes a language model on the combined training data
+for Indian banking transaction entity extraction.
+
+Supports:
+- MLX (Apple Silicon) via mlx-lm
+- PyTorch/Transformers (GPU/CPU)
+
+Usage:
+    python finetune.py --model microsoft/Phi-3-mini-4k-instruct --epochs 3
+"""
+
+import json
+import argparse
+import os
+import sys
+from pathlib import Path
+from datetime import datetime
+from typing import List, Dict, Optional, Tuple
+import random
+
+# Check for MLX (Apple Silicon)
+try:
+    import mlx
+    import mlx.core as mx
+    HAS_MLX = True
+except ImportError:
+    HAS_MLX = False
+
+# Check for PyTorch
+try:
+    import torch
+    HAS_TORCH = True
+except ImportError:
+    HAS_TORCH = False
+
+
+# ============================================================================
+# DATA PREPARATION
+# ============================================================================
+
+class DataPreparer:
+    """Prepare training data for fine-tuning."""
+    
+    SYSTEM_PROMPT = """You are a finance entity extraction assistant for Indian banking. 
+Extract structured information from banking SMS/email messages.
+
+Output JSON with these fields (only include if found):
+- amount: float (transaction amount)
+- type: "debit" or "credit"
+- account: string (last 4 digits)
+- bank: string (bank name)
+- date: string (transaction date)
+- reference: string (UPI/NEFT reference)
+- merchant: string (business name for P2M)
+- beneficiary: string (person name for P2P)
+- vpa: string (UPI ID)
+- category: string (food, shopping, travel, etc.)
+- is_p2m: boolean (true if merchant, false if person)
+
+Be precise. Extract exactly what's in the message."""
+
+    def __init__(self, data_path: Path, val_split: float = 0.1):
+        self.data_path = data_path
+        self.val_split = val_split
+        self.train_data = []
+        self.val_data = []
+    
+    def load_and_split(self) -> Tuple[List[Dict], List[Dict]]:
+        """Load data and split into train/val."""
+        print(f"Loading data from {self.data_path}...")
+        
+        all_data = []
+        with open(self.data_path, 'r', encoding='utf-8') as f:
+            for line in f:
+                try:
+                    record = json.loads(line)
+                    all_data.append(record)
+                except json.JSONDecodeError:
+                    continue
+        
+        print(f"  Loaded {len(all_data):,} records")
+        
+        # Shuffle
+        random.shuffle(all_data)
+        
+        # Split
+        split_idx = int(len(all_data) * (1 - self.val_split))
+        self.train_data = all_data[:split_idx]
+        self.val_data = all_data[split_idx:]
+        
+        print(f"  Train: {len(self.train_data):,}, Val: {len(self.val_data):,}")
+        
+        return self.train_data, self.val_data
+    
+    def format_for_chat(self, record: Dict) -> Dict:
+        """Format record for chat-style fine-tuning."""
+        input_text = record.get('input', record.get('text', ''))
+        output_text = record.get('output', '{}')
+        
+        if isinstance(output_text, dict):
+            output_text = json.dumps(output_text, ensure_ascii=False)
+        
+        return {
+            'messages': [
+                {'role': 'system', 'content': self.SYSTEM_PROMPT},
+                {'role': 'user', 'content': input_text},
+                {'role': 'assistant', 'content': output_text},
+            ]
+        }
+    
+    def format_for_completion(self, record: Dict) -> Dict:
+        """Format record for completion-style fine-tuning."""
+        input_text = record.get('input', record.get('text', ''))
+        output_text = record.get('output', '{}')
+        
+        if isinstance(output_text, dict):
+            output_text = json.dumps(output_text, ensure_ascii=False)
+        
+        prompt = f"""Extract financial entities from this message:
+
+Message: {input_text}
+
+JSON:"""
+        
+        return {
+            'prompt': prompt,
+            'completion': output_text,
+        }
+    
+    def save_formatted(
+        self,
+        output_dir: Path,
+        format_type: str = 'chat'
+    ) -> Tuple[Path, Path]:
+        """Save formatted train/val data."""
+        output_dir.mkdir(parents=True, exist_ok=True)
+        
+        train_path = output_dir / 'train.jsonl'
+        val_path = output_dir / 'valid.jsonl'
+        
+        formatter = (
+            self.format_for_chat if format_type == 'chat'
+            else self.format_for_completion
+        )
+        
+        # Save train
+        with open(train_path, 'w', encoding='utf-8') as f:
+            for record in self.train_data:
+                formatted = formatter(record)
+                f.write(json.dumps(formatted, ensure_ascii=False) + '\n')
+        
+        # Save val
+        with open(val_path, 'w', encoding='utf-8') as f:
+            for record in self.val_data:
+                formatted = formatter(record)
+                f.write(json.dumps(formatted, ensure_ascii=False) + '\n')
+        
+        print(f"  Saved train: {train_path}")
+        print(f"  Saved valid: {val_path}")
+        
+        return train_path, val_path
+
+
+# ============================================================================
+# MLX FINE-TUNING (Apple Silicon)
+# ============================================================================
+
+class MLXFineTuner:
+    """Fine-tune using MLX-LM on Apple Silicon."""
+    
+    def __init__(
+        self,
+        model_name: str,
+        output_dir: Path,
+        lora_rank: int = 8,
+        lora_layers: int = 16,
+    ):
+        self.model_name = model_name
+        self.output_dir = output_dir
+        self.lora_rank = lora_rank
+        self.lora_layers = lora_layers
+    
+    def train(
+        self,
+        train_path: Path,
+        val_path: Path,
+        epochs: int = 3,
+        batch_size: int = 4,
+        learning_rate: float = 1e-5,
+        save_every: int = 100,
+    ):
+        """Run MLX-LM LoRA fine-tuning."""
+        import subprocess
+        
+        cmd = [
+            sys.executable, '-m', 'mlx_lm.lora',
+            '--model', self.model_name,
+            '--train',
+            '--data', str(train_path.parent),
+            '--lora-layers', str(self.lora_layers),
+            '--lora-rank', str(self.lora_rank),
+            '--batch-size', str(batch_size),
+            '--iters', str(epochs * 1000),
+            '--learning-rate', str(learning_rate),
+            '--save-every', str(save_every),
+            '--adapter-path', str(self.output_dir / 'adapters'),
+        ]
+        
+        print(f"\n🚀 Starting MLX-LM LoRA training...")
+        print(f"   Command: {' '.join(cmd)}")
+        print()
+        
+        result = subprocess.run(cmd, capture_output=False)
+        
+        return result.returncode == 0
+    
+    def fuse(self):
+        """Fuse LoRA adapters with base model."""
+        import subprocess
+        
+        adapter_path = self.output_dir / 'adapters'
+        fused_path = self.output_dir / 'fused'
+        
+        cmd = [
+            sys.executable, '-m', 'mlx_lm.fuse',
+            '--model', self.model_name,
+            '--adapter-path', str(adapter_path),
+            '--save-path', str(fused_path),
+        ]
+        
+        print(f"\n🔗 Fusing LoRA adapters...")
+        result = subprocess.run(cmd, capture_output=False)
+        
+        return result.returncode == 0
+
+
+# ============================================================================
+# PYTORCH/TRANSFORMERS FINE-TUNING
+# ============================================================================
+
+class TransformersFineTuner:
+    """Fine-tune using PyTorch/Transformers."""
+    
+    def __init__(
+        self,
+        model_name: str,
+        output_dir: Path,
+        lora_rank: int = 8,
+    ):
+        self.model_name = model_name
+        self.output_dir = output_dir
+        self.lora_rank = lora_rank
+    
+    def train(
+        self,
+        train_path: Path,
+        val_path: Path,
+        epochs: int = 3,
+        batch_size: int = 4,
+        learning_rate: float = 2e-5,
+    ):
+        """Run Transformers fine-tuning with PEFT."""
+        try:
+            from transformers import (
+                AutoModelForCausalLM,
+                AutoTokenizer,
+                TrainingArguments,
+                Trainer,
+                DataCollatorForSeq2Seq,
+            )
+            from peft import LoraConfig, get_peft_model
+            from datasets import load_dataset
+        except ImportError as e:
+            print(f"❌ Missing dependencies: {e}")
+            print("   Run: pip install transformers peft datasets")
+            return False
+        
+        print(f"\n🚀 Loading model: {self.model_name}")
+        
+        # Load model & tokenizer
+        tokenizer = AutoTokenizer.from_pretrained(self.model_name)
+        model = AutoModelForCausalLM.from_pretrained(
+            self.model_name,
+            torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
+            device_map='auto' if torch.cuda.is_available() else None,
+        )
+        
+        # Add padding token if needed
+        if tokenizer.pad_token is None:
+            tokenizer.pad_token = tokenizer.eos_token
+        
+        # LoRA config
+        lora_config = LoraConfig(
+            r=self.lora_rank,
+            lora_alpha=32,
+            target_modules=['q_proj', 'v_proj', 'k_proj', 'o_proj'],
+            lora_dropout=0.05,
+            bias='none',
+            task_type='CAUSAL_LM',
+        )
+        
+        model = get_peft_model(model, lora_config)
+        model.print_trainable_parameters()
+        
+        # Load dataset
+        dataset = load_dataset(
+            'json',
+            data_files={
+                'train': str(train_path),
+                'validation': str(val_path),
+            }
+        )
+        
+        # Tokenize
+        def tokenize(examples):
+            # For chat format
+            if 'messages' in examples:
+                texts = []
+                for msgs in examples['messages']:
+                    text = ''
+                    for msg in msgs:
+                        text += f"<|{msg['role']}|>\n{msg['content']}\n"
+                    texts.append(text)
+            else:
+                texts = [f"{p}\n{c}" for p, c in zip(examples['prompt'], examples['completion'])]
+            
+            return tokenizer(
+                texts,
+                truncation=True,
+                max_length=512,
+                padding='max_length',
+            )
+        
+        tokenized = dataset.map(tokenize, batched=True, remove_columns=dataset['train'].column_names)
+        
+        # Training args
+        training_args = TrainingArguments(
+            output_dir=str(self.output_dir),
+            num_train_epochs=epochs,
+            per_device_train_batch_size=batch_size,
+            per_device_eval_batch_size=batch_size,
+            learning_rate=learning_rate,
+            logging_steps=100,
+            save_steps=500,
+            evaluation_strategy='steps',
+            eval_steps=500,
+            fp16=torch.cuda.is_available(),
+            report_to='none',
+        )
+        
+        # Trainer
+        trainer = Trainer(
+            model=model,
+            args=training_args,
+            train_dataset=tokenized['train'],
+            eval_dataset=tokenized['validation'],
+            data_collator=DataCollatorForSeq2Seq(tokenizer, padding=True),
+        )
+        
+        print(f"\n🚀 Starting training...")
+        trainer.train()
+        
+        # Save
+        model.save_pretrained(self.output_dir / 'adapters')
+        tokenizer.save_pretrained(self.output_dir / 'adapters')
+        
+        print(f"\n✅ Saved to: {self.output_dir / 'adapters'}")
+        return True
+
+
+# ============================================================================
+# EVALUATION
+# ============================================================================
+
+class Evaluator:
+    """Evaluate fine-tuned model."""
+    
+    def __init__(self, model_path: Path, backend: str = 'mlx'):
+        self.model_path = model_path
+        self.backend = backend
+    
+    def evaluate(self, test_data: List[Dict], max_samples: int = 100) -> Dict:
+        """Evaluate on test data."""
+        if self.backend == 'mlx':
+            return self._evaluate_mlx(test_data[:max_samples])
+        else:
+            return self._evaluate_torch(test_data[:max_samples])
+    
+    def _evaluate_mlx(self, test_data: List[Dict]) -> Dict:
+        """Evaluate with MLX."""
+        from mlx_lm import load, generate
+        
+        model, tokenizer = load(str(self.model_path))
+        
+        correct = 0
+        total = 0
+        field_matches = {'amount': 0, 'type': 0, 'merchant': 0}
+        
+        for record in test_data:
+            input_text = record.get('input', record.get('text', ''))
+            expected = record.get('output', '{}')
+            if isinstance(expected, str):
+                expected = json.loads(expected)
+            
+            prompt = f"Extract financial entities:\n\n{input_text}\n\nJSON:"
+            
+            output = generate(
+                model, tokenizer, prompt,
+                max_tokens=256,
+                temp=0.0,
+            )
+            
+            try:
+                predicted = json.loads(output)
+                
+                # Check fields
+                for field in field_matches:
+                    if predicted.get(field) == expected.get(field):
+                        field_matches[field] += 1
+                
+                # Full match
+                if predicted == expected:
+                    correct += 1
+            except json.JSONDecodeError:
+                pass
+            
+            total += 1
+        
+        return {
+            'accuracy': correct / total if total > 0 else 0,
+            'field_accuracy': {k: v/total for k, v in field_matches.items()},
+            'total_samples': total,
+        }
+    
+    def _evaluate_torch(self, test_data: List[Dict]) -> Dict:
+        """Evaluate with PyTorch."""
+        from transformers import AutoModelForCausalLM, AutoTokenizer
+        from peft import PeftModel
+        
+        # Load
+        base_model = AutoModelForCausalLM.from_pretrained(
+            self.model_path.parent / 'base',
+            torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
+        )
+        model = PeftModel.from_pretrained(base_model, str(self.model_path))
+        tokenizer = AutoTokenizer.from_pretrained(str(self.model_path))
+        
+        # Similar evaluation logic...
+        return {'accuracy': 0, 'note': 'PyTorch evaluation not fully implemented'}
+
+
+# ============================================================================
+# MAIN PIPELINE
+# ============================================================================
+
+def main():
+    parser = argparse.ArgumentParser(description="LLM Fine-tuning Pipeline")
+    parser.add_argument('--data', default='data/training/final_combined_training.jsonl',
+                       help='Training data path')
+    parser.add_argument('--model', default='microsoft/Phi-3-mini-4k-instruct',
+                       help='Base model')
+    parser.add_argument('--output', default='models/finetune',
+                       help='Output directory')
+    parser.add_argument('--epochs', type=int, default=3, help='Training epochs')
+    parser.add_argument('--batch-size', type=int, default=4, help='Batch size')
+    parser.add_argument('--lr', type=float, default=1e-5, help='Learning rate')
+    parser.add_argument('--lora-rank', type=int, default=8, help='LoRA rank')
+    parser.add_argument('--backend', choices=['mlx', 'torch', 'auto'], default='auto',
+                       help='Training backend')
+    parser.add_argument('--skip-train', action='store_true', help='Skip training, just prepare data')
+    parser.add_argument('--evaluate', action='store_true', help='Evaluate after training')
+    
+    args = parser.parse_args()
+    
+    # Determine backend
+    if args.backend == 'auto':
+        if HAS_MLX:
+            backend = 'mlx'
+            print("🍎 Using MLX (Apple Silicon)")
+        elif HAS_TORCH:
+            backend = 'torch'
+            print("🔥 Using PyTorch/Transformers")
+        else:
+            print("❌ No backend available. Install mlx-lm or transformers+peft")
+            return
+    else:
+        backend = args.backend
+    
+    data_path = Path(args.data)
+    output_dir = Path(args.output)
+    timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
+    run_dir = output_dir / f"run_{timestamp}"
+    
+    print("=" * 60)
+    print("🚀 LLM FINE-TUNING PIPELINE")
+    print("=" * 60)
+    print(f"  Data: {data_path}")
+    print(f"  Model: {args.model}")
+    print(f"  Output: {run_dir}")
+    print(f"  Backend: {backend}")
+    print(f"  Epochs: {args.epochs}")
+    
+    # Step 1: Prepare data
+    print("\n📋 Step 1: Preparing data...")
+    preparer = DataPreparer(data_path)
+    train_data, val_data = preparer.load_and_split()
+    
+    formatted_dir = run_dir / 'data'
+    format_type = 'chat' if backend == 'torch' else 'completion'
+    train_path, val_path = preparer.save_formatted(formatted_dir, format_type)
+    
+    if args.skip_train:
+        print("\n⏭️ Skipping training (--skip-train)")
+        return
+    
+    # Step 2: Train
+    print("\n🎯 Step 2: Training...")
+    
+    if backend == 'mlx':
+        trainer = MLXFineTuner(
+            model_name=args.model,
+            output_dir=run_dir,
+            lora_rank=args.lora_rank,
+        )
+        success = trainer.train(
+            train_path, val_path,
+            epochs=args.epochs,
+            batch_size=args.batch_size,
+            learning_rate=args.lr,
+        )
+        
+        if success:
+            print("\n🔗 Step 3: Fusing adapters...")
+            trainer.fuse()
+    else:
+        trainer = TransformersFineTuner(
+            model_name=args.model,
+            output_dir=run_dir,
+            lora_rank=args.lora_rank,
+        )
+        success = trainer.train(
+            train_path, val_path,
+            epochs=args.epochs,
+            batch_size=args.batch_size,
+            learning_rate=args.lr,
+        )
+    
+    # Step 3: Evaluate
+    if args.evaluate and success:
+        print("\n📊 Step 4: Evaluating...")
+        evaluator = Evaluator(run_dir / 'fused' if backend == 'mlx' else run_dir / 'adapters', backend)
+        results = evaluator.evaluate(val_data)
+        
+        print(f"\n📊 Results:")
+        print(f"  Overall Accuracy: {results.get('accuracy', 0):.1%}")
+        for field, acc in results.get('field_accuracy', {}).items():
+            print(f"  {field}: {acc:.1%}")
+        
+        # Save results
+        with open(run_dir / 'eval_results.json', 'w') as f:
+            json.dump(results, f, indent=2)
+    
+    print("\n" + "=" * 60)
+    print("✅ FINE-TUNING COMPLETE")
+    print("=" * 60)
+    print(f"  Output: {run_dir}")
+    print(f"  Adapters: {run_dir / 'adapters'}")
+    if backend == 'mlx':
+        print(f"  Fused model: {run_dir / 'fused'}")
+
+
+if __name__ == "__main__":
+    main()