docs/synthetic_data_guide.md

# Synthetic Data Generation - Methodology & Design

## Overview

This document describes the methodology, reasons, and approach used to generate **realistic synthetic data** for the Metro Train Scheduling System. The synthetic data mimics real-world KMRL (Kochi Metro Rail Limited) operational patterns and constraints.

---

## Table of Contents

1. [Why Synthetic Data?](#why-synthetic-data)
2. [Design Principles](#design-principles)
3. [Generation Methodology](#generation-methodology)
4. [Data Schema](#data-schema)
5. [Realistic Patterns & Distributions](#realistic-patterns--distributions)
6. [Validation & Quality Assurance](#validation--quality-assurance)

---

## Why Synthetic Data?

### Reasons for Synthetic Data Generation

**1. Privacy & Compliance**
- Real metro operational data contains sensitive information
- Cannot expose actual train maintenance issues or financial data
- Protects commercial partnerships (advertising contracts)
- Avoids regulatory compliance issues

**2. Development & Testing**
- No access to production KMRL data during development
- Need large volumes of data for ML model training (100+ schedules)
- Requires controlled data for testing edge cases
- Enables reproducible experiments

**3. Demonstration & Validation**
- Showcase system capabilities without real data dependencies
- Create demo scenarios for stakeholders
- Test algorithm performance under various conditions
- Validate optimization quality metrics

**4. Scalability**
- Generate data for different fleet sizes (25-40 trains)
- Create scenarios with varying operational constraints
- Simulate different time periods and seasons
- Model edge cases rarely seen in production

**5. Cost Efficiency**
- No data acquisition costs
- No data cleaning/preprocessing overhead
- Immediate availability for development
- Can generate on-demand for specific test cases

---

## Design Principles

### 1. **Realism**
Generate data that closely mirrors actual metro operations:
- Real station names from KMRL Aluva-Pettah Line
- Actual distance (25.612 km) and station count (25)
- Realistic operational hours (5 AM - 11 PM)
- Industry-standard maintenance patterns

### 2. **Statistical Distribution**
Model real-world probabilities:
- 65% trains fully healthy
- 20% partially available (limited hours)
- 15% unavailable (maintenance/breakdown)
- Normal distribution for mileage, readiness scores

### 3. **Consistency**
Maintain logical relationships:
- High mileage → lower readiness scores
- More job cards → higher maintenance probability
- Expired certificates → unavailable status
- Maintenance history affects current health

### 4. **Variability**
Introduce realistic randomness:
- Different fitness certificate expiry dates
- Varying branding contracts and priorities
- Random maintenance windows
- Stochastic component failures

### 5. **Constraint Adherence**
Respect operational rules:
- Minimum service trains (22-24)
- Minimum standby capacity (3-5)
- Depot capacity limits
- Turnaround time requirements

---

## Generation Methodology

### Class: `MetroDataGenerator`
**Location**: `DataService/metro_data_generator.py`

### Step-by-Step Generation Process

#### 1. Route Generation
```python
def generate_route():
    # Use real KMRL stations
    stations = ["Aluva", "Pulinchodu", ..., "Pettah"]  # 25 stations
    total_distance = 25.612 km  # Actual KMRL distance
    
    for each station:
        - Calculate distance from origin (linear interpolation)
        - Assign dwell time (20-45 seconds, random)
        - Set sequence number
    
    return Route with:
        - avg_speed: 32-38 km/h (realistic metro speed)
        - turnaround_time: 8-12 minutes (standard metro practice)
```

**Reasoning**:
- Real station names → authentic demonstration
- Linear distance → simplified but representative
- Random dwell times → models station complexity variation
- Speed range → typical metro performance

---

#### 2. Train Health Status Generation
```python
def generate_train_health_statuses():
    for each train:
        health_roll = random(0, 1)
        
        if health_roll < 0.65:  # 65% probability
            status = "Fully Healthy"
            available_hours = None  # Available all operational hours
        
        elif health_roll < 0.85:  # 20% probability
            status = "Partially Healthy"
            available_hours = random window (e.g., 5 AM - 2 PM)
            reason = "Minor repairs" | "Partial maintenance"
        
        else:  # 15% probability
            status = "Unavailable"
            available_hours = []
            reason = random choice from:
                - SCHEDULED_MAINTENANCE
                - BRAKE_SYSTEM_REPAIR
                - HVAC_REPLACEMENT
                - BOGIE_OVERHAUL
                - ELECTRICAL_FAULT
                - ACCIDENT_DAMAGE
                - PANTOGRAPH_REPAIR
                - DOOR_SYSTEM_FAULT
```

**Reasoning**:
- **65% healthy**: Most trains operational (industry standard ~70%)
- **20% partial**: Common in metros with aging fleet or scheduled maintenance
- **15% unavailable**: Realistic for daily maintenance needs (2-4 trains in 30-train fleet)
- **Specific reasons**: Real maintenance categories for authenticity

**Distribution Logic**:
```
Fleet size = 30 trains
├── Fully Healthy: 19-20 trains (can serve all day)
├── Partially Healthy: 6 trains (limited availability)
└── Unavailable: 4-5 trains (in maintenance/repair)
```

---

#### 3. Fitness Certificates Generation
```python
def generate_fitness_certificates(train_id):
    certificates = {
        "rolling_stock": generate_certificate(),
        "signalling": generate_certificate(),
        "telecom": generate_certificate()
    }
    
def generate_certificate():
    roll = random(0, 1)
    
    if roll < 0.70:  # 70% valid
        expiry_date = today + random(45, 365) days
        status = VALID
    
    elif roll < 0.90:  # 20% expiring soon
        expiry_date = today + random(7, 30) days
        status = EXPIRING_SOON
    
    else:  # 10% expired
        expiry_date = today - random(1, 30) days
        status = EXPIRED
```

**Reasoning**:
- **3 certificate types**: Regulatory requirement for metro safety
- **70% valid**: Most trains compliant (good operational health)
- **20% expiring soon**: Warning system for proactive renewal
- **10% expired**: Reflects renewal process delays (realistic bureaucracy)

**Impact on Scheduling**:
- EXPIRED → Train status = UNAVAILABLE (hard constraint)
- EXPIRING_SOON → Flagged in alerts, can still operate (soft constraint)
- VALID → No impact on scheduling

---

#### 4. Job Cards (Maintenance Tracking)
```python
def generate_job_cards(train_id):
    num_open_cards = weighted_random([0, 1, 2, 3, 4, 5])
    weights = [50%, 25%, 15%, 7%, 2%, 1%]
    
    blocking_issues = []
    if num_open_cards > 0:
        # Some job cards are "blocking" (critical)
        if random() < 0.3:  # 30% chance
            blocking_issues.append(random choice from critical_faults)
    
    return JobCards(
        open=num_open_cards,
        blocking=blocking_issues
    )
```

**Reasoning**:
- **Most trains (50%)**: No open job cards (well-maintained)
- **25%**: 1 job card (minor issue)
- **15%**: 2 job cards (moderate maintenance)
- **Decreasing probability**: Reflects good maintenance practices
- **Blocking issues**: Critical faults that prevent operation

**Impact on Readiness**:
```python
readiness_score = base_readiness * (1 - 0.1 * num_open_cards)
0 cards → 1.0 readiness
1 card  → 0.9 readiness
2 cards → 0.8 readiness
5 cards → 0.5 readiness (likely in maintenance)
```

---

#### 5. Branding & Advertisement
```python
def generate_branding():
    advertiser = random choice from:
        - COCACOLA-2024
        - FLIPKART-FESTIVE
        - AMAZON-PRIME
        - RELIANCE-JIO
        - TATA-MOTORS
        - SAMSUNG-GALAXY
        - NONE (50% probability)
    
    if advertiser != "NONE":
        contract_hours_remaining = random(50, 500)
        exposure_priority = random choice:
            - LOW (40%)
            - MEDIUM (30%)
            - HIGH (20%)
            - CRITICAL (10%)
    else:
        contract_hours_remaining = 0
        exposure_priority = "NONE"
```

**Reasoning**:
- **50% no branding**: Half the fleet has no ads (realistic for public transport)
- **50% branded**: Active advertising contracts
- **Real brand names**: Examples of typical advertisers (FMCG, tech, retail)
- **Priority levels**: Different SLA requirements based on contract value

**Scheduling Impact**:
- HIGH/CRITICAL branded trains prioritized for peak hours
- Maximizes passenger exposure → higher advertiser ROI
- Adds revenue optimization objective to schedule

---

#### 6. Mileage Distribution
```python
def get_realistic_mileage_distribution(num_trains):
    # Target average: 150,000 km (5-7 years of operation)
    # Standard deviation: 20,000 km (variation in usage)
    
    base_mileages = normal_distribution(
        mean=150000,
        std=20000,
        size=num_trains
    )
    
    # Add age-based clustering
    # 30% newer trains (100k-130k)
    # 50% mid-life trains (130k-170k)
    # 20% older trains (170k-200k)
    
    return clipped(base_mileages, min=80000, max=220000)
```

**Reasoning**:
- **Normal distribution**: Natural wear pattern over time
- **Mean 150,000 km**: Typical for 5-7 year old fleet
- **Clustering**: Reflects batch procurement (trains bought in groups)
- **Variance**: Different usage patterns (some trains used more than others)

**Impact**:
- High mileage → lower priority (balance wear across fleet)
- Mileage variance → optimization objective (minimize imbalance)

---

#### 7. Readiness Score Calculation
```python
def calculate_readiness_score(train):
    score = 1.0  # Start at perfect
    
    # Factor 1: Certificate status (-30% if expired)
    if any_certificate_expired:
        score *= 0.0  # Cannot operate
    elif any_certificate_expiring_soon:
        score *= 0.85  # Minor penalty
    
    # Factor 2: Job cards (-10% per card)
    score *= (1.0 - 0.1 * num_open_job_cards)
    
    # Factor 3: Component health (average of all components)
    score *= average(component_health_scores)
    
    # Factor 4: Time since last major maintenance
    days_since_maintenance = (today - last_major_service).days
    if days_since_maintenance > 90:
        score *= 0.9  # Needs service soon
    
    # Factor 5: Age/mileage penalty
    if mileage > 180000:
        score *= 0.95
    
    return max(0.0, min(1.0, score))
```

**Reasoning**:
- **Multi-factor assessment**: Holistic train health evaluation
- **Hard constraints**: Expired certificates → score = 0
- **Soft degradation**: Accumulating issues gradually reduce score
- **Realistic range**: Most trains score 0.7-0.95
- **Bounded [0,1]**: Normalized for optimization algorithms

---

#### 8. Depot & Bay Assignment
```python
DEPOT_BAYS = ["BAY-01", "BAY-02", ..., "BAY-15"]  # 15 parking bays
IBL_BAYS = ["IBL-01", ..., "IBL-05"]  # 5 inspection bays
WASH_BAYS = ["WASH-BAY-01", "WASH-BAY-02", "WASH-BAY-03"]

def assign_depot_bay(train_status):
    if train_status == "REVENUE_SERVICE":
        return "IN-SERVICE"  # Not at depot
    
    elif train_status == "STANDBY":
        return random choice from DEPOT_BAYS
    
    elif train_status == "MAINTENANCE":
        # 70% in regular bay, 30% in inspection bay
        if random() < 0.7:
            return random choice from DEPOT_BAYS
        else:
            return random choice from IBL_BAYS
    
    elif train_status == "CLEANING":
        return random choice from WASH_BAYS
```

**Reasoning**:
- **15 depot bays**: Typical for 25-30 train fleet (some trains in service)
- **5 IBL (Inspection) bays**: Specialized maintenance facilities
- **3 wash bays**: Limited washing capacity (bottleneck)
- **Random assignment**: Simulates dynamic depot management

---

## Data Schema

### Generated Synthetic Data Structures

#### 1. Route Schema
```json
{
  "route_id": "KMRL-LINE-01",
  "name": "Aluva-Pettah Line",
  "stations": [
    {
      "station_id": "STN-001",
      "name": "Aluva",
      "sequence": 1,
      "distance_from_origin_km": 0.0,
      "avg_dwell_time_seconds": 35
    },
    ...
  ],
  "total_distance_km": 25.612,
  "avg_speed_kmh": 35,
  "turnaround_time_minutes": 10
}
```

**Size**: ~5 KB (25 stations)

---

#### 2. Train Health Status Schema
```json
{
  "trainset_id": "TS-001",
  "is_healthy": true,
  "available_hours": null,
  "reason": null
}
```

**Variations**:
```json
// Partially healthy
{
  "trainset_id": "TS-015",
  "is_healthy": false,
  "available_hours": [
    ["05:00", "14:00"]  // Available 5 AM - 2 PM only
  ],
  "reason": "Minor repairs - limited service window"
}

// Unavailable
{
  "trainset_id": "TS-023",
  "is_healthy": false,
  "available_hours": [],
  "reason": "BRAKE_SYSTEM_REPAIR"
}
```

**Size**: ~150 bytes per train

---

#### 3. Fitness Certificates Schema
```json
{
  "rolling_stock": {
    "valid_until": "2026-03-15",
    "status": "VALID"
  },
  "signalling": {
    "valid_until": "2025-12-20",
    "status": "EXPIRING_SOON"
  },
  "telecom": {
    "valid_until": "2025-10-01",
    "status": "EXPIRED"
  }
}
```

**Status Values**:
- `VALID`: > 30 days remaining
- `EXPIRING_SOON`: 7-30 days remaining
- `EXPIRED`: Past expiry date

**Size**: ~200 bytes per train

---

#### 4. Job Cards Schema
```json
{
  "open": 2,
  "blocking": ["BRAKE_FAULT", "DOOR_MALFUNCTION"]
}
```

**Blocking Issues** (Critical):
- BRAKE_FAULT
- POWER_FAILURE
- COUPLING_DEFECT
- SAFETY_SYSTEM_ERROR
- STRUCTURAL_DAMAGE

**Size**: ~100 bytes per train

---

#### 5. Branding Schema
```json
{
  "advertiser": "COCACOLA-2024",
  "contract_hours_remaining": 245,
  "exposure_priority": "HIGH"
}
```

**Priority Mapping**:
- CRITICAL: 4 points (highest exposure requirement)
- HIGH: 3 points
- MEDIUM: 2 points
- LOW: 1 point
- NONE: 0 points (no advertiser)

**Size**: ~80 bytes per train

---

#### 6. Component Health Schema
```json
{
  "brakes": 0.92,
  "hvac": 0.88,
  "doors": 0.95,
  "bogies": 0.87,
  "pantograph": 0.90,
  "electrical": 0.93,
  "communication": 0.89
}
```

**Range**: [0.0, 1.0]
- 0.95-1.0: Excellent condition
- 0.85-0.95: Good condition
- 0.70-0.85: Fair condition (may need service soon)
- < 0.70: Poor condition (maintenance required)

**Size**: ~150 bytes per train

---

#### 7. Mileage Data Schema
```json
{
  "trainset_id": "TS-012",
  "cumulative_km": 145250,
  "last_service_km": 142000,
  "next_service_due_km": 150000,
  "daily_average_km": 285
}
```

**Typical Values**:
- New trains: 80,000 - 120,000 km
- Mid-life: 120,000 - 170,000 km
- Older: 170,000 - 220,000 km
- Daily average: 250-350 km (varies by assignment)

**Size**: ~120 bytes per train

---

### Complete Trainset Data Example

```json
{
  "trainset_id": "TS-012",
  "status": "REVENUE_SERVICE",
  "depot_bay": "IN-SERVICE",
  "cumulative_km": 145250,
  "readiness_score": 0.87,
  "service_blocks": [
    {
      "block_id": "BLK-012-01",
      "start_time": "05:30",
      "end_time": "06:15",
      "start_station": "Aluva",
      "end_station": "Pettah",
      "direction": "DOWN",
      "distance_km": 25.612
    },
    ...
  ],
  "fitness_certificates": {
    "rolling_stock": {"valid_until": "2026-02-15", "status": "VALID"},
    "signalling": {"valid_until": "2025-12-10", "status": "EXPIRING_SOON"},
    "telecom": {"valid_until": "2026-01-20", "status": "VALID"}
  },
  "job_cards": {
    "open": 1,
    "blocking": []
  },
  "branding": {
    "advertiser": "SAMSUNG-GALAXY",
    "contract_hours_remaining": 187,
    "exposure_priority": "MEDIUM"
  },
  "component_health": {
    "brakes": 0.92,
    "hvac": 0.85,
    "doors": 0.94,
    "bogies": 0.88,
    "pantograph": 0.91,
    "electrical": 0.90,
    "communication": 0.87
  }
}
```

**Total Size**: ~1.5 KB per trainset

---

## Realistic Patterns & Distributions

### 1. Health Status Distribution

```
30-train fleet expected distribution:

Fully Healthy (65%):        ████████████████████  19-20 trains
Partially Available (20%):  ██████                6 trains
Unavailable (15%):          ████                  4-5 trains
```

### 2. Certificate Status Distribution

```
Per certificate type (90 total certificates for 30 trains):

VALID (70%):           ██████████████████████  63 certificates
EXPIRING_SOON (20%):   ██████                   18 certificates
EXPIRED (10%):         ███                      9 certificates
```

### 3. Job Card Distribution

```
30-train fleet:

0 open cards (50%):  ███████████████  15 trains  (excellent)
1 open card (25%):   ███████          7-8 trains (good)
2 open cards (15%):  ████             4-5 trains (fair)
3+ cards (10%):      ███              3 trains   (needs attention)
```

### 4. Branding Distribution

```
Advertiser assignment:

NONE (50%):          ███████████████  15 trains
COCACOLA (8%):       ██               2-3 trains
FLIPKART (8%):       ██               2-3 trains
AMAZON (8%):         ██               2-3 trains
Others (26%):        ███████          7-8 trains
```

```
Priority distribution (branded trains only):

LOW (40%):       ██████         6 trains
MEDIUM (30%):    ████           4-5 trains
HIGH (20%):      ███            3 trains
CRITICAL (10%):  █              1-2 trains
```

### 5. Readiness Score Distribution

```
Expected distribution (histogram):

0.95-1.00 (Excellent):  ███████       7 trains   (25%)
0.85-0.95 (Good):       ████████████  12 trains  (40%)
0.70-0.85 (Fair):       ████████      8 trains   (27%)
0.50-0.70 (Poor):       ██            2 trains   (7%)
< 0.50 (Critical):      █             1 train    (3%)
```

**Mean**: 0.84  
**Median**: 0.87  
**Std Dev**: 0.12

---

## Validation & Quality Assurance

### Automated Validation Checks

#### 1. **Constraint Validation**
```python
def validate_generated_data(data):
    assert len(data.trainsets) == num_trains
    assert all(0 <= t.readiness_score <= 1.0 for t in trainsets)
    assert sum(t.status == "REVENUE_SERVICE") >= min_service_trains
    assert sum(t.status == "STANDBY") >= min_standby_trains
```

#### 2. **Distribution Testing**
```python
# Test health status distribution
healthy_count = count(status == "healthy")
assert 0.60 <= healthy_count / total <= 0.70  # Should be ~65%

# Test certificate validity
expired_count = count(certificates == "EXPIRED")
assert 0.08 <= expired_count / total_certs <= 0.12  # Should be ~10%
```

#### 3. **Logical Consistency**
```python
# Expired certificates → Unavailable status
for train in trainsets:
    if any_certificate_expired(train):
        assert train.status != "REVENUE_SERVICE"

# Blocking job cards → Maintenance/Unavailable
for train in trainsets:
    if len(train.job_cards.blocking) > 0:
        assert train.status in ["MAINTENANCE", "UNAVAILABLE"]
```

#### 4. **Statistical Tests**
```python
# Mileage distribution (Shapiro-Wilk test for normality)
mileages = [t.cumulative_km for t in trainsets]
statistic, p_value = shapiro(mileages)
assert p_value > 0.05  # Accept null hypothesis (normal distribution)

# Readiness scores (mean should be around 0.85)
mean_readiness = mean([t.readiness_score for t in trainsets])
assert 0.80 <= mean_readiness <= 0.90
```

---

## Usage in System

### 1. **Initial Training Data Generation**
```python
# Generate 150 schedules for ML training
for i in range(150):
    generator = MetroDataGenerator(num_trains=25 + (i % 15))
    route = generator.generate_route()
    health_statuses = generator.generate_train_health_statuses()
    
    # ... generate schedule and save
```

### 2. **API Request Handling**
```python
@app.post("/api/v1/generate")
def generate_schedule(request):
    generator = MetroDataGenerator(
        num_trains=request.num_trains,
        num_stations=request.num_stations
    )
    
    # Generate fresh synthetic data for this request
    route = generator.generate_route()
    health = generator.generate_train_health_statuses()
    
    # Optimize schedule with synthetic data
    schedule = optimize(route, health, ...)
    return schedule
```

### 3. **Testing & Benchmarking**
```python
# Generate edge case scenarios
scenarios = {
    "high_maintenance": lambda: set_maintenance_rate(0.30),
    "certificate_crisis": lambda: set_expiry_rate(0.25),
    "low_availability": lambda: set_healthy_rate(0.50)
}

for name, scenario in scenarios.items():
    data = generate_synthetic_data_with(scenario)
    result = optimize(data)
    assert result.feasible
```

---

## Limitations & Future Enhancements

### Current Limitations

1. **Static Patterns**: Health status doesn't evolve over time
2. **Independent Generation**: Each train generated independently (no fleet-wide correlations)
3. **Simplified Geography**: Linear distance interpolation (doesn't model actual track layout)
4. **No Seasonality**: Doesn't model seasonal variations (monsoon, festivals)
5. **No Historical Trends**: Doesn't consider past schedules or performance

### Planned Enhancements

1. **Time-Series Generation**: Model degradation over days/weeks
2. **Correlated Failures**: If one train has HVAC issue, higher probability for others
3. **GIS Integration**: Use actual station coordinates and track geometry
4. **Event Modeling**: Special events, holidays, peak seasons
5. **Historical Patterns**: Learn from past schedules to generate more realistic data
6. **Real Data Validation**: Compare synthetic data distributions with actual KMRL data (when available)

---

## Summary

### Key Takeaways

✅ **Realistic Distributions**: 65/20/15 health split mirrors industry norms  
✅ **Multi-Factor Modeling**: Readiness considers certificates, maintenance, age  
✅ **Logical Consistency**: Expired certificates → unavailable status  
✅ **Statistical Rigor**: Normal distributions for mileage, validated ranges  
✅ **Operational Authenticity**: Real station names, actual distances, realistic speeds  
✅ **Comprehensive Coverage**: Covers all aspects (health, certificates, branding, maintenance)  
✅ **Validation Built-in**: Automated checks ensure data quality  

**Total Synthetic Data per Schedule**: ~48 KB (30 trains)  
**Generation Time**: < 0.5 seconds  
**Validation Pass Rate**: > 99%  

---

**Document Version**: 1.0.0  
**Last Updated**: November 4, 2025  
**Maintained By**: DataService Team