Deploy to Hugging Face Spaces: Add application files and dependencies

- Add FastAPI application (app.py)
- Add preprocessing module (preprocessing.py)
- Add model utilities (model_utils.py)
- Add Dockerfile for HF Spaces deployment
- Add requirements.txt
- Add deployment documentation
- Add configuration files

GIT_COMMANDS.md

@@ -0,0 +1,193 @@
+# คำสั่ง Git สำหรับ Deploy ไปยัง Hugging Face Spaces
+
+## วิธีที่ 1: ใช้ Script (แนะนำ)
+
+### Windows (PowerShell)
+```powershell
+.\deploy.ps1
+```
+
+### Linux/Mac (Bash)
+```bash
+chmod +x deploy.sh
+./deploy.sh
+```
+
+---
+
+## วิธีที่ 2: รันคำสั่ง Git แบบ Manual
+
+### Step 1: ลบ Git Lock File (ถ้ามี)
+```powershell
+# PowerShell
+Remove-Item .git\index.lock -Force -ErrorAction SilentlyContinue
+```
+
+```bash
+# Bash
+rm -f .git/index.lock
+```
+
+### Step 2: Add ไฟล์ Core Application
+```bash
+git add app.py
+git add preprocessing.py
+git add model_utils.py
+git add requirements.txt
+git add Dockerfile
+```
+
+### Step 3: Add Configuration Files
+```bash
+git add .dockerignore
+git add .gitignore
+```
+
+### Step 4: Add Documentation
+```bash
+git add README_HF_SPACE.md
+git add DEPLOYMENT.md
+git add DEPLOYMENT_SUMMARY.md
+git add README.md
+```
+
+### Step 5: Add Models Directory (ถ้าต้องการ)
+```bash
+# ถ้า models/ ไม่อยู่ใน .gitignore
+git add models/
+
+# ถ้า models/ อยู่ใน .gitignore แต่ต้องการ commit
+git add -f models/
+```
+
+### Step 6: ตรวจสอบ Files ที่จะ Commit
+```bash
+git status --short
+```
+
+### Step 7: Commit
+```bash
+git commit -m "Deploy to Hugging Face Spaces: Add application files and dependencies
+
+- Add FastAPI application (app.py)
+- Add preprocessing module (preprocessing.py)
+- Add model utilities (model_utils.py)
+- Add Dockerfile for HF Spaces deployment
+- Add requirements.txt
+- Add deployment documentation
+- Add configuration files"
+```
+
+### Step 8: Push
+```bash
+git push
+```
+
+หรือ
+
+```bash
+git push origin main
+```
+
+---
+
+## วิธีที่ 3: คำสั่งเดียว (Quick)
+
+```bash
+# Add ไฟล์ทั้งหมดที่จำเป็น
+git add app.py preprocessing.py model_utils.py requirements.txt Dockerfile .dockerignore .gitignore README_HF_SPACE.md DEPLOYMENT.md DEPLOYMENT_SUMMARY.md README.md
+
+# Add models (ถ้าต้องการ)
+git add models/ || git add -f models/
+
+# Commit
+git commit -m "Deploy to Hugging Face Spaces: Add application files and dependencies"
+
+# Push
+git push
+```
+
+---
+
+## ไฟล์ที่ต้อง Commit
+
+### ✅ ไฟล์ที่จำเป็น (Required)
+- `app.py` - FastAPI application
+- `preprocessing.py` - Preprocessing module
+- `model_utils.py` - Model utilities
+- `requirements.txt` - Python dependencies
+- `Dockerfile` - Docker configuration
+
+### ✅ ไฟล์ Configuration
+- `.dockerignore` - Docker ignore rules
+- `.gitignore` - Git ignore rules
+
+### ✅ ไฟล์ Documentation
+- `README_HF_SPACE.md` - README for HF Space
+- `DEPLOYMENT.md` - Deployment guide
+- `DEPLOYMENT_SUMMARY.md` - Deployment summary
+- `README.md` - Main README
+
+### ⚠️ Models Directory
+- `models/` - Model files (ต้อง commit ถ้าต้องการใช้ local models)
+
+---
+
+## Troubleshooting
+
+### ปัญหา: Git Lock File
+```bash
+# ลบ lock file
+Remove-Item .git\index.lock -Force  # PowerShell
+rm -f .git/index.lock               # Bash
+```
+
+### ปัญหา: Authentication Required
+```bash
+# ตั้งค่า credential
+git config --global credential.helper store
+
+# หรือใช้ token
+git remote set-url origin https://[token]@huggingface.co/spaces/[username]/[space-name]
+```
+
+### ปัญหา: Models ไม่ถูก Commit
+```bash
+# Force add models directory
+git add -f models/
+```
+
+### ปัญหา: Remote URL ไม่ถูกต้อง
+```bash
+# ตรวจสอบ remote URL
+git remote -v
+
+# ตั้งค่า remote URL (ถ้าจำเป็น)
+git remote set-url origin https://huggingface.co/spaces/[username]/[space-name]
+```
+
+---
+
+## Checklist
+
+- [ ] ลบ git lock file (ถ้ามี)
+- [ ] Add ไฟล์ core application
+- [ ] Add configuration files
+- [ ] Add documentation files
+- [ ] Add models directory (ถ้าต้องการ)
+- [ ] ตรวจสอบ git status
+- [ ] Commit พร้อม message
+- [ ] Push ไปยัง remote
+- [ ] ตรวจสอบ build logs ใน Hugging Face Spaces
+
+---
+
+## หมายเหตุ
+
+1. **Models Directory**: ถ้า `models/` อยู่ใน `.gitignore` ต้องใช้ `git add -f models/` เพื่อ force add
+
+2. **Authentication**: Hugging Face Spaces ต้องใช้ token หรือ credential สำหรับ push
+
+3. **Build Time**: หลัง push แล้ว Hugging Face Spaces จะ rebuild อัตโนมัติ (~5-15 นาที)
+
+4. **File Size**: ตรวจสอบขนาด models ถ้าใหญ่เกิน 1GB อาจต้องใช้ Git LFS

SETUP.md

@@ -0,0 +1,128 @@
+# Setup Instructions
+
+## 1. Install Dependencies
+
+### Option A: Using pip (recommended)
+
+```bash
+pip install -r requirements.txt
+```
+
+### Option B: Using virtual environment (recommended for isolation)
+
+```bash
+# Create virtual environment
+python -m venv venv
+
+# Activate virtual environment
+# On Windows (PowerShell):
+.\venv\Scripts\Activate.ps1
+# On Windows (CMD):
+venv\Scripts\activate.bat
+# On Linux/Mac:
+source venv/bin/activate
+
+# Install dependencies
+pip install -r requirements.txt
+```
+
+## 2. Prepare Data
+
+Place your CSV files in the `data/` directory:
+
+```bash
+mkdir -p data
+# Copy your CSV files to data/
+```
+
+## 3. Train Models
+
+### Train Job Failure Prediction Model
+
+```bash
+# Option 1: Use all CSV files in data/ directory (recommended)
+python train_job_failure.py
+
+# Option 2: Specify files explicitly
+python train_job_failure.py data/true_export_report_20260120.csv data/true_export_report_20260121.csv
+
+# Option 3: Use glob pattern (works in PowerShell and bash)
+python train_job_failure.py data/*.csv
+```
+
+### Train Anomaly Detection Model
+
+```bash
+# Option 1: Use all CSV files in data/ directory (recommended)
+python train_anomaly.py
+
+# Option 2: Specify files explicitly
+python train_anomaly.py data/true_export_report_20260120.csv data/true_export_report_20260121.csv
+
+# Option 3: Use glob pattern
+python train_anomaly.py data/*.csv
+```
+
+## 4. Verify Models
+
+After training, check that models were created:
+
+```bash
+ls models/
+```
+
+You should see:
+- `job_fail_pipeline_cpu.joblib`
+- `anomaly_autoencoder_cpu.keras`
+- `anomaly_scaler.joblib`
+- `feature_schema.json`
+- `shap_background.npy`
+- `anomaly_features.joblib`
+- `anomaly_threshold.joblib`
+
+## 5. Run the Service
+
+### Local Development
+
+```bash
+uvicorn app:app --host 0.0.0.0 --port 8000
+```
+
+### Docker
+
+```bash
+docker-compose up --build
+```
+
+## Troubleshooting
+
+### "ModuleNotFoundError: No module named 'tensorflow'"
+
+Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+### "No data files found!"
+
+1. Check that CSV files exist in `data/` directory:
+   ```bash
+   ls data/
+   ```
+
+2. Use explicit file paths:
+   ```bash
+   python train_job_failure.py data/true_export_report_20260120.csv
+   ```
+
+### PowerShell glob pattern not working
+
+The scripts now handle glob patterns internally. You can use:
+```powershell
+python train_job_failure.py data/*.csv
+```
+
+Or just run without arguments to use all CSV files in `data/`:
+```powershell
+python train_job_failure.py
+```

app.py

@@ -4,7 +4,7 @@ FastAPI application for Job Failure Prediction and Anomaly Detection.
 from fastapi import FastAPI, HTTPException
 from fastapi.middleware.cors import CORSMiddleware
 from pydantic import BaseModel, Field
-from typing import Optional, Dict, List
+from typing import Optional, Dict, List, Any
 import pandas as pd
 from datetime import datetime
 import os
@@ -75,7 +75,7 @@ class JobFailureResponse(BaseModel):
     """Response model for job failure prediction."""
     fail_probability: float
     risk_level: str
-    top_drivers: Optional[List[Dict[str, any]]] = None
+    top_drivers: Optional[List[Dict[str, Any]]] = None
     recommended_actions: Optional[List[str]] = None
     error: Optional[str] = None
 
@@ -96,7 +96,7 @@ class AnomalyResponse(BaseModel):
     reconstruction_error: float
     is_anomaly: bool
     threshold: float
-    top_drivers: List[Dict[str, any]]
+    top_drivers: List[Dict[str, Any]]
     error: Optional[str] = None
 
 
@@ -188,4 +188,4 @@ async def root():
 
 if __name__ == "__main__":
     import uvicorn
-    uvicorn.run(app, host="0.0.0.0", port=8000)
+    uvicorn.run(app, host="0.0.0.0", port=7860)

deploy.ps1

@@ -0,0 +1,93 @@
+# Git deployment script for Hugging Face Spaces
+# This script adds, commits, and pushes all necessary files
+
+Write-Host "=== Preparing files for Hugging Face Spaces deployment ===" -ForegroundColor Cyan
+
+# Change to project directory
+Set-Location $PSScriptRoot
+
+# Step 1: Remove git lock file if exists (in case of previous failed operations)
+if (Test-Path .git\index.lock) {
+    Write-Host "Removing git lock file..." -ForegroundColor Yellow
+    Remove-Item .git\index.lock -Force -ErrorAction SilentlyContinue
+}
+
+# Step 2: Add core application files (required for API to work)
+Write-Host "`nAdding core application files..." -ForegroundColor Green
+git add app.py
+git add preprocessing.py
+git add model_utils.py
+git add requirements.txt
+git add Dockerfile
+
+# Step 3: Add configuration files
+Write-Host "Adding configuration files..." -ForegroundColor Green
+git add .dockerignore
+git add .gitignore
+
+# Step 4: Add documentation files
+Write-Host "Adding documentation files..." -ForegroundColor Green
+git add README_HF_SPACE.md
+git add DEPLOYMENT.md
+git add DEPLOYMENT_SUMMARY.md
+git add README.md
+
+# Step 5: Add models directory (if exists and not in .gitignore)
+if (Test-Path models) {
+    Write-Host "Checking models directory..." -ForegroundColor Yellow
+    # Check if models/ is in .gitignore
+    $gitignoreContent = Get-Content .gitignore -ErrorAction SilentlyContinue
+    if ($gitignoreContent -notmatch "^models/") {
+        Write-Host "Adding models directory..." -ForegroundColor Green
+        git add models/
+    } else {
+        Write-Host "WARNING: models/ is in .gitignore. You may need to force add it:" -ForegroundColor Yellow
+        Write-Host "  git add -f models/" -ForegroundColor Yellow
+    }
+}
+
+# Step 6: Show status
+Write-Host "`n=== Files staged for commit ===" -ForegroundColor Cyan
+git status --short
+
+# Step 7: Commit
+Write-Host "`n=== Committing changes ===" -ForegroundColor Cyan
+$commitMessage = "Deploy to Hugging Face Spaces: Add application files and dependencies
+
+- Add FastAPI application (app.py)
+- Add preprocessing module (preprocessing.py)
+- Add model utilities (model_utils.py)
+- Add Dockerfile for HF Spaces deployment
+- Add requirements.txt
+- Add deployment documentation
+- Add configuration files"
+
+git commit -m $commitMessage
+
+if ($LASTEXITCODE -eq 0) {
+    Write-Host "`n=== Commit successful! ===" -ForegroundColor Green
+    
+    # Step 8: Push
+    Write-Host "`n=== Pushing to remote ===" -ForegroundColor Cyan
+    Write-Host "Note: You may need to authenticate for push" -ForegroundColor Yellow
+    git push
+    
+    if ($LASTEXITCODE -eq 0) {
+        Write-Host "`n=== Deployment files pushed successfully! ===" -ForegroundColor Green
+        Write-Host "Next steps:" -ForegroundColor Cyan
+        Write-Host "1. Go to your Hugging Face Space" -ForegroundColor White
+        Write-Host "2. Wait for automatic rebuild" -ForegroundColor White
+        Write-Host "3. Check logs if build fails" -ForegroundColor White
+    } else {
+        Write-Host "`n=== Push failed ===" -ForegroundColor Red
+        Write-Host "You may need to:" -ForegroundColor Yellow
+        Write-Host "1. Set up authentication (git credential or token)" -ForegroundColor White
+        Write-Host "2. Check remote URL: git remote -v" -ForegroundColor White
+        Write-Host "3. Try: git push origin main" -ForegroundColor White
+    }
+} else {
+    Write-Host "`n=== Commit failed ===" -ForegroundColor Red
+    Write-Host "Check git status and resolve any conflicts" -ForegroundColor Yellow
+}
+
+Write-Host "`n=== Script completed ===" -ForegroundColor Cyan

deploy.sh

@@ -0,0 +1,96 @@
+#!/bin/bash
+# Git deployment script for Hugging Face Spaces
+# This script adds, commits, and pushes all necessary files
+
+echo "=== Preparing files for Hugging Face Spaces deployment ==="
+
+# Step 1: Remove git lock file if exists
+if [ -f .git/index.lock ]; then
+    echo "Removing git lock file..."
+    rm -f .git/index.lock
+fi
+
+# Step 2: Add core application files (required for API to work)
+echo ""
+echo "Adding core application files..."
+git add app.py
+git add preprocessing.py
+git add model_utils.py
+git add requirements.txt
+git add Dockerfile
+
+# Step 3: Add configuration files
+echo "Adding configuration files..."
+git add .dockerignore
+git add .gitignore
+
+# Step 4: Add documentation files
+echo "Adding documentation files..."
+git add README_HF_SPACE.md
+git add DEPLOYMENT.md
+git add DEPLOYMENT_SUMMARY.md
+git add README.md
+
+# Step 5: Add models directory (if exists and not in .gitignore)
+if [ -d models ]; then
+    echo "Checking models directory..."
+    if ! grep -q "^models/" .gitignore 2>/dev/null; then
+        echo "Adding models directory..."
+        git add models/
+    else
+        echo "WARNING: models/ is in .gitignore. You may need to force add it:"
+        echo "  git add -f models/"
+    fi
+fi
+
+# Step 6: Show status
+echo ""
+echo "=== Files staged for commit ==="
+git status --short
+
+# Step 7: Commit
+echo ""
+echo "=== Committing changes ==="
+git commit -m "Deploy to Hugging Face Spaces: Add application files and dependencies
+
+- Add FastAPI application (app.py)
+- Add preprocessing module (preprocessing.py)
+- Add model utilities (model_utils.py)
+- Add Dockerfile for HF Spaces deployment
+- Add requirements.txt
+- Add deployment documentation
+- Add configuration files"
+
+if [ $? -eq 0 ]; then
+    echo ""
+    echo "=== Commit successful! ==="
+    
+    # Step 8: Push
+    echo ""
+    echo "=== Pushing to remote ==="
+    echo "Note: You may need to authenticate for push"
+    git push
+    
+    if [ $? -eq 0 ]; then
+        echo ""
+        echo "=== Deployment files pushed successfully! ==="
+        echo "Next steps:"
+        echo "1. Go to your Hugging Face Space"
+        echo "2. Wait for automatic rebuild"
+        echo "3. Check logs if build fails"
+    else
+        echo ""
+        echo "=== Push failed ==="
+        echo "You may need to:"
+        echo "1. Set up authentication (git credential or token)"
+        echo "2. Check remote URL: git remote -v"
+        echo "3. Try: git push origin main"
+    fi
+else
+    echo ""
+    echo "=== Commit failed ==="
+    echo "Check git status and resolve any conflicts"
+fi
+
+echo ""
+echo "=== Script completed ==="

docker-compose.yml

@@ -0,0 +1,30 @@
+version: '3.8'
+
+services:
+  job-ml:
+    build:
+      context: .
+      dockerfile: Dockerfile
+    container_name: job-ml-service
+    ports:
+      - "8000:8000"
+    volumes:
+      # Mount models directory so models persist and can be updated
+      - ./models:/app/models
+      # Mount data directory for training data (optional)
+      - ./data:/app/data
+    environment:
+      - PYTHONUNBUFFERED=1
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "python", "-c", "import requests; requests.get('http://localhost:8000/health')"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
+    networks:
+      - ml-network
+
+networks:
+  ml-network:
+    driver: bridge

n8n_workflow_examples.md

@@ -0,0 +1,336 @@
+# n8n Workflow Examples for Job Failure Prediction
+
+This document provides examples for integrating the ML service with n8n workflows.
+
+## Service Endpoints
+
+- **Base URL**: `http://job-ml:8000` (internal) or `http://localhost:8000` (local)
+- **Health Check**: `GET /health`
+- **Job Failure Prediction**: `POST /predict/job-fail`
+- **Anomaly Detection**: `POST /detect/anomaly`
+
+## Example 1: Job Failure Prediction Workflow
+
+### Workflow Structure
+1. **Trigger**: Database query / Webhook / Schedule
+2. **HTTP Request**: Call prediction endpoint
+3. **IF Node**: Check risk level
+4. **Action**: Send alert (Slack / PagerDuty / Email)
+
+### HTTP Request Node Configuration
+
+**Method**: `POST`  
+**URL**: `http://job-ml:8000/predict/job-fail`  
+**Headers**:
+```
+Content-Type: application/json
+```
+
+**Body (JSON)**:
+```json
+{
+  "zone": "{{ $json.zone }}",
+  "job_nm": "{{ $json.job_nm }}",
+  "tasksgroup_nm": "{{ $json.tasksgroup_nm }}",
+  "job_start_time": "{{ $json.job_start_time }}",
+  "duration": "{{ $json.duration }}",
+  "duration_sec": {{ $json.duration_sec }},
+  "status": "{{ $json.status }}",
+  "err_msg": "{{ $json.err_msg }}",
+  "zeppelin": "{{ $json.zeppelin }}",
+  "explain": true
+}
+```
+
+### IF Node Conditions
+
+**WARNING Condition** (Medium Risk):
+```javascript
+{{ $json.fail_probability >= 0.5 && $json.fail_probability < 0.8 }}
+```
+
+**CRITICAL Condition** (High Risk):
+```javascript
+{{ $json.fail_probability >= 0.8 }}
+```
+
+**Combined Alert Condition** (WARNING or CRITICAL):
+```javascript
+{{ $json.fail_probability >= 0.5 }}
+```
+
+### Example Response Handling
+
+The response will look like:
+```json
+{
+  "fail_probability": 0.79,
+  "risk_level": "MEDIUM",
+  "top_drivers": [
+    {
+      "feature": "failure_rate_7",
+      "shap_value": 0.30,
+      "effect": "increase"
+    },
+    {
+      "feature": "duration_zscore",
+      "shap_value": 0.18,
+      "effect": "increase"
+    }
+  ],
+  "recommended_actions": [
+    "Monitor upstream dependencies and recent job history",
+    "Check for resource constraints or data volume spikes"
+  ]
+}
+```
+
+### Slack Alert Example
+
+**Slack Node Configuration**:
+- **Channel**: `#job-alerts`
+- **Text**:
+```
+🚨 Job Failure Alert
+
+Job: {{ $json.job_nm }}
+Zone: {{ $json.zone }}
+Risk Level: *{{ $('HTTP Request').item.json.risk_level }}*
+Failure Probability: {{ $('HTTP Request').item.json.fail_probability * 100 }}%
+
+Top Risk Factors:
+{{ $('HTTP Request').item.json.top_drivers.map(d => `• ${d.feature}: ${d.effect}`).join('\n') }}
+
+Recommended Actions:
+{{ $('HTTP Request').item.json.recommended_actions.map(a => `• ${a}`).join('\n') }}
+```
+
+---
+
+## Example 2: Anomaly Detection Workflow
+
+### HTTP Request Node Configuration
+
+**Method**: `POST`  
+**URL**: `http://job-ml:8000/detect/anomaly`  
+**Headers**:
+```
+Content-Type: application/json
+```
+
+**Body (JSON)**:
+```json
+{
+  "features": {
+    "duration_sec": {{ $json.duration_sec }},
+    "duration_zscore": {{ $json.duration_zscore }},
+    "avg_duration_7": {{ $json.avg_duration_7 }},
+    "failure_rate_7": {{ $json.failure_rate_7 }},
+    "err_msg_len": {{ $json.err_msg_len || 0 }},
+    "hour_sin": {{ $json.hour_sin || 0 }},
+    "hour_cos": {{ $json.hour_cos || 0 }}
+  },
+  "threshold": 0.01
+}
+```
+
+### IF Node Condition
+
+**Anomaly Detected**:
+```javascript
+{{ $json.is_anomaly === true }}
+```
+
+**High Severity Anomaly** (reconstruction error > 3x threshold):
+```javascript
+{{ $json.is_anomaly === true && $json.reconstruction_error > ($json.threshold * 3) }}
+```
+
+### Example Response
+
+```json
+{
+  "reconstruction_error": 0.0235,
+  "is_anomaly": true,
+  "threshold": 0.01,
+  "top_drivers": [
+    {
+      "feature": "duration_zscore",
+      "error": 0.0142
+    },
+    {
+      "feature": "duration_sec",
+      "error": 0.0068
+    }
+  ]
+}
+```
+
+---
+
+## Example 3: Combined Workflow (Prediction + Anomaly)
+
+### Workflow Structure
+1. **Trigger**: Job completion event
+2. **HTTP Request 1**: Job failure prediction
+3. **HTTP Request 2**: Anomaly detection
+4. **IF Node**: Combined alert logic
+5. **Action**: Send alert
+
+### Combined Alert Condition
+
+**Alert if EITHER prediction is risky OR anomaly is detected**:
+```javascript
+{{ 
+  ($('Predict').item.json.fail_probability >= 0.5) || 
+  ($('Anomaly').item.json.is_anomaly === true) 
+}}
+```
+
+**High Priority Alert** (both conditions):
+```javascript
+{{
+  ($('Predict').item.json.fail_probability >= 0.8) ||
+  ($('Anomaly').item.json.is_anomaly === true && $('Anomaly').item.json.reconstruction_error > ($('Anomaly').item.json.threshold * 3))
+}}
+```
+
+---
+
+## Example 4: Scheduled Monitoring Workflow
+
+### Workflow Structure
+1. **Schedule Trigger**: Every 15 minutes
+2. **Database Query**: Get recent job runs
+3. **Loop**: For each job
+   - HTTP Request: Predict failure
+   - HTTP Request: Detect anomaly
+   - IF: Check conditions
+   - Action: Alert if needed
+
+### Code Node for Batch Processing
+
+```javascript
+// Process multiple jobs
+const jobs = $input.all();
+const results = [];
+
+for (const job of jobs) {
+  // Call prediction API
+  const predictResponse = await $http.post('http://job-ml:8000/predict/job-fail', {
+    zone: job.json.zone,
+    job_nm: job.json.job_nm,
+    job_start_time: job.json.job_start_time,
+    duration_sec: job.json.duration_sec,
+    status: job.json.status,
+    err_msg: job.json.err_msg,
+    explain: true
+  });
+  
+  // Call anomaly API
+  const anomalyResponse = await $http.post('http://job-ml:8000/detect/anomaly', {
+    features: {
+      duration_sec: job.json.duration_sec,
+      duration_zscore: job.json.duration_zscore || 0,
+      err_msg_len: (job.json.err_msg || '').length
+    }
+  });
+  
+  results.push({
+    job: job.json.job_nm,
+    prediction: predictResponse.data,
+    anomaly: anomalyResponse.data,
+    should_alert: predictResponse.data.fail_probability >= 0.5 || anomalyResponse.data.is_anomaly
+  });
+}
+
+return results.map(r => ({ json: r }));
+```
+
+---
+
+## Alert Severity Levels
+
+### INFO
+- `fail_probability < 0.3` and no anomaly
+- Normal operations
+
+### LOW
+- `0.3 <= fail_probability < 0.5`
+- Minor deviations detected
+
+### WARNING (MEDIUM)
+- `0.5 <= fail_probability < 0.8`
+- OR `is_anomaly === true` with `reconstruction_error < threshold * 3`
+- Requires monitoring
+
+### CRITICAL
+- `fail_probability >= 0.8`
+- OR `is_anomaly === true` with `reconstruction_error >= threshold * 3`
+- Immediate action required
+
+---
+
+## Error Handling
+
+### HTTP Request Error Handling
+
+In n8n, configure the HTTP Request node to:
+- **Continue on Error**: Enabled
+- **Response Format**: JSON
+
+Add an IF node after HTTP Request to check for errors:
+```javascript
+{{ $json.error !== undefined && $json.error !== null }}
+```
+
+### Retry Logic
+
+For critical predictions, add a retry mechanism:
+1. HTTP Request node
+2. IF node: Check if response has error
+3. Wait node: 5 seconds
+4. HTTP Request node: Retry (max 3 times)
+
+---
+
+## Testing the Workflow
+
+### Test Payload (Job Failure Prediction)
+
+```json
+{
+  "zone": "prod",
+  "job_nm": "daily_export_customer",
+  "tasksgroup_nm": "export_group",
+  "job_start_time": "2026-01-21T01:00:00",
+  "duration": "01:30:00",
+  "duration_sec": 5400,
+  "status": "SUCCESS",
+  "err_msg": "",
+  "zeppelin": null
+}
+```
+
+### Test Payload (Anomaly Detection)
+
+```json
+{
+  "features": {
+    "duration_sec": 5400,
+    "duration_zscore": 1.6,
+    "avg_duration_7": 3000,
+    "failure_rate_7": 0.15,
+    "err_msg_len": 0,
+    "hour_sin": 0.2588,
+    "hour_cos": 0.9659
+  },
+  "threshold": 0.01
+}
+```
+
+---
+
+## n8n Workflow JSON Export
+
+See `n8n_workflow_job_monitoring.json` for a complete importable workflow.

n8n_workflow_job_monitoring.json

@@ -0,0 +1,224 @@
+{
+  "name": "Job Failure Monitoring",
+  "nodes": [
+    {
+      "parameters": {
+        "rule": {
+          "interval": [
+            {
+              "field": "minutes",
+              "minutesInterval": 15
+            }
+          ]
+        }
+      },
+      "id": "schedule-trigger",
+      "name": "Schedule Trigger",
+      "type": "n8n-nodes-base.scheduleTrigger",
+      "typeVersion": 1.1,
+      "position": [250, 300]
+    },
+    {
+      "parameters": {
+        "method": "POST",
+        "url": "http://job-ml:8000/predict/job-fail",
+        "authentication": "none",
+        "sendHeaders": true,
+        "headerParameters": {
+          "parameters": [
+            {
+              "name": "Content-Type",
+              "value": "application/json"
+            }
+          ]
+        },
+        "sendBody": true,
+        "bodyParameters": {
+          "parameters": []
+        },
+        "specifyBody": "json",
+        "jsonBody": "={\n  \"zone\": \"{{ $json.zone }}\",\n  \"job_nm\": \"{{ $json.job_nm }}\",\n  \"tasksgroup_nm\": \"{{ $json.tasksgroup_nm }}\",\n  \"job_start_time\": \"{{ $json.job_start_time }}\",\n  \"duration\": \"{{ $json.duration }}\",\n  \"duration_sec\": {{ $json.duration_sec }},\n  \"status\": \"{{ $json.status }}\",\n  \"err_msg\": \"{{ $json.err_msg }}\",\n  \"zeppelin\": \"{{ $json.zeppelin }}\",\n  \"explain\": true\n}",
+        "options": {}
+      },
+      "id": "predict-http",
+      "name": "Predict Job Failure",
+      "type": "n8n-nodes-base.httpRequest",
+      "typeVersion": 4.1,
+      "position": [450, 300]
+    },
+    {
+      "parameters": {
+        "conditions": {
+          "options": {
+            "caseSensitive": true,
+            "leftValue": "",
+            "typeValidation": "strict"
+          },
+          "conditions": [
+            {
+              "id": "risk-check",
+              "leftValue": "={{ $json.fail_probability }}",
+              "rightValue": 0.5,
+              "operator": {
+                "type": "number",
+                "operation": "gte"
+              }
+            }
+          ],
+          "combinator": "and"
+        },
+        "options": {}
+      },
+      "id": "risk-if",
+      "name": "Check Risk Level",
+      "type": "n8n-nodes-base.if",
+      "typeVersion": 2,
+      "position": [650, 300]
+    },
+    {
+      "parameters": {
+        "method": "POST",
+        "url": "http://job-ml:8000/detect/anomaly",
+        "authentication": "none",
+        "sendHeaders": true,
+        "headerParameters": {
+          "parameters": [
+            {
+              "name": "Content-Type",
+              "value": "application/json"
+            }
+          ]
+        },
+        "sendBody": true,
+        "specifyBody": "json",
+        "jsonBody": "={\n  \"features\": {\n    \"duration_sec\": {{ $json.duration_sec }},\n    \"duration_zscore\": {{ $json.duration_zscore || 0 }},\n    \"err_msg_len\": {{ ($json.err_msg || '').length }}\n  },\n  \"threshold\": 0.01\n}",
+        "options": {}
+      },
+      "id": "anomaly-http",
+      "name": "Detect Anomaly",
+      "type": "n8n-nodes-base.httpRequest",
+      "typeVersion": 4.1,
+      "position": [650, 200]
+    },
+    {
+      "parameters": {
+        "conditions": {
+          "options": {
+            "caseSensitive": true,
+            "leftValue": "",
+            "typeValidation": "strict"
+          },
+          "conditions": [
+            {
+              "id": "anomaly-check",
+              "leftValue": "={{ $json.is_anomaly }}",
+              "rightValue": true,
+              "operator": {
+                "type": "boolean",
+                "operation": "true"
+              }
+            }
+          ],
+          "combinator": "and"
+        },
+        "options": {}
+      },
+      "id": "anomaly-if",
+      "name": "Check Anomaly",
+      "type": "n8n-nodes-base.if",
+      "typeVersion": 2,
+      "position": [850, 200]
+    },
+    {
+      "parameters": {
+        "channel": "#job-alerts",
+        "text": "=🚨 Job Failure Alert\n\nJob: {{ $('Predict Job Failure').item.json.job_nm }}\nZone: {{ $('Predict Job Failure').item.json.zone }}\nRisk Level: *{{ $('Predict Job Failure').item.json.risk_level }}*\nFailure Probability: {{ Math.round($('Predict Job Failure').item.json.fail_probability * 100) }}%\n\nTop Risk Factors:\n{{ $('Predict Job Failure').item.json.top_drivers.map(d => `• ${d.feature}: ${d.effect}`).join('\\n') }}\n\nRecommended Actions:\n{{ $('Predict Job Failure').item.json.recommended_actions.map(a => `• ${a}`).join('\\n') }}",
+        "otherOptions": {}
+      },
+      "id": "slack-alert",
+      "name": "Send Slack Alert",
+      "type": "n8n-nodes-base.slack",
+      "typeVersion": 2.1,
+      "position": [1050, 300],
+      "credentials": {
+        "slackApi": {
+          "id": "slack-credentials",
+          "name": "Slack API"
+        }
+      }
+    }
+  ],
+  "connections": {
+    "Schedule Trigger": {
+      "main": [
+        [
+          {
+            "node": "Predict Job Failure",
+            "type": "main",
+            "index": 0
+          }
+        ]
+      ]
+    },
+    "Predict Job Failure": {
+      "main": [
+        [
+          {
+            "node": "Check Risk Level",
+            "type": "main",
+            "index": 0
+          }
+        ]
+      ]
+    },
+    "Check Risk Level": {
+      "main": [
+        [
+          {
+            "node": "Send Slack Alert",
+            "type": "main",
+            "index": 0
+          }
+        ],
+        [
+          {
+            "node": "Detect Anomaly",
+            "type": "main",
+            "index": 0
+          }
+        ]
+      ]
+    },
+    "Detect Anomaly": {
+      "main": [
+        [
+          {
+            "node": "Check Anomaly",
+            "type": "main",
+            "index": 0
+          }
+        ]
+      ]
+    },
+    "Check Anomaly": {
+      "main": [
+        [
+          {
+            "node": "Send Slack Alert",
+            "type": "main",
+            "index": 0
+          }
+        ]
+      ]
+    }
+  },
+  "pinData": {},
+  "settings": {
+    "executionOrder": "v1"
+  },
+  "staticData": null,
+  "tags": [],
+  "triggerCount": 1,
+  "updatedAt": "2026-01-25T00:00:00.000Z",
+  "versionId": "1"
+}

requirements.txt

@@ -1,12 +1,11 @@
-fastapi==0.104.1
-uvicorn[standard]==0.24.0
-pydantic==2.5.0
-pandas==2.1.3
-numpy==1.24.3
-scikit-learn>=1.3.0,<1.4.0
-xgboost>=2.1.0
-tensorflow==2.15.0
-shap==0.43.0
-joblib==1.3.2
-python-multipart==0.0.6
-requests==2.31.0
+fastapi
+uvicorn[standard]
+pydantic
+numpy
+scikit-learn
+xgboost
+tensorflow
+shap
+joblib
+python-multipart
+requests

test_api.py

@@ -0,0 +1,79 @@
+"""
+Simple test script for API endpoints.
+Run this after starting the service to verify endpoints work correctly.
+"""
+import requests
+import json
+
+BASE_URL = "http://localhost:8000"
+
+
+def test_health():
+    """Test health endpoint."""
+    print("Testing /health endpoint...")
+    response = requests.get(f"{BASE_URL}/health")
+    print(f"Status: {response.status_code}")
+    print(f"Response: {json.dumps(response.json(), indent=2)}")
+    print()
+
+
+def test_predict():
+    """Test job failure prediction endpoint."""
+    print("Testing /predict/job-fail endpoint...")
+    payload = {
+        "zone": "prod",
+        "job_nm": "daily_export_customer",
+        "tasksgroup_nm": "export_group",
+        "job_start_time": "2026-01-21T01:00:00",
+        "duration": "01:30:00",
+        "duration_sec": 5400,
+        "status": "SUCCESS",
+        "err_msg": "",
+        "zeppelin": None,
+        "explain": True
+    }
+    response = requests.post(
+        f"{BASE_URL}/predict/job-fail",
+        json=payload,
+        headers={"Content-Type": "application/json"}
+    )
+    print(f"Status: {response.status_code}")
+    print(f"Response: {json.dumps(response.json(), indent=2)}")
+    print()
+
+
+def test_anomaly():
+    """Test anomaly detection endpoint."""
+    print("Testing /detect/anomaly endpoint...")
+    payload = {
+        "features": {
+            "duration_sec": 5400,
+            "duration_zscore": 1.6,
+            "avg_duration_7": 3000,
+            "failure_rate_7": 0.15,
+            "err_msg_len": 0,
+            "hour_sin": 0.2588,
+            "hour_cos": 0.9659
+        },
+        "threshold": 0.01
+    }
+    response = requests.post(
+        f"{BASE_URL}/detect/anomaly",
+        json=payload,
+        headers={"Content-Type": "application/json"}
+    )
+    print(f"Status: {response.status_code}")
+    print(f"Response: {json.dumps(response.json(), indent=2)}")
+    print()
+
+
+if __name__ == "__main__":
+    try:
+        test_health()
+        test_predict()
+        test_anomaly()
+        print("All tests completed!")
+    except requests.exceptions.ConnectionError:
+        print("Error: Could not connect to API. Make sure the service is running on http://localhost:8000")
+    except Exception as e:
+        print(f"Error: {e}")

train_anomaly.py

@@ -0,0 +1,183 @@
+"""
+Training script for Anomaly Detection using Autoencoder.
+"""
+import pandas as pd
+import numpy as np
+from sklearn.preprocessing import StandardScaler
+from tensorflow import keras
+from tensorflow.keras import Model, Input
+from tensorflow.keras.layers import Dense
+import joblib
+import os
+from preprocessing import engineer_features, get_feature_columns
+
+
+def train_anomaly_detector(
+    train_csv_paths: list,
+    output_dir: str = 'models',
+    encoding_dim: int = 32,
+    epochs: int = 50,
+    batch_size: int = 256,
+    random_state: int = 42
+):
+    """
+    Train autoencoder for anomaly detection.
+    
+    Args:
+        train_csv_paths: List of CSV file paths to load
+        output_dir: Directory to save model artifacts
+        encoding_dim: Dimension of encoding layer
+        epochs: Training epochs
+        batch_size: Batch size
+        random_state: Random seed
+    """
+    os.makedirs(output_dir, exist_ok=True)
+    np.random.seed(random_state)
+    
+    # Load and combine data
+    print("Loading data...")
+    dfs = []
+    for csv_path in train_csv_paths:
+        if os.path.exists(csv_path):
+            df = pd.read_csv(csv_path)
+            dfs.append(df)
+            print(f"  Loaded {len(df)} rows from {csv_path}")
+        else:
+            print(f"  Warning: {csv_path} not found, skipping")
+    
+    if not dfs:
+        raise ValueError("No data files found!")
+    
+    df = pd.concat(dfs, ignore_index=True)
+    print(f"Total rows: {len(df)}")
+    
+    # Engineer features
+    print("Engineering features...")
+    df = engineer_features(df)
+    
+    # Get numeric features for autoencoder
+    feature_cols = get_feature_columns()
+    num_cols = feature_cols['anomaly_numeric']
+    
+    # Ensure all columns exist
+    for col in num_cols:
+        if col not in df.columns:
+            df[col] = 0.0
+    
+    X = df[num_cols].copy()
+    X = X.fillna(0.0)
+    
+    print(f"Anomaly features: {num_cols}")
+    print(f"Feature matrix shape: {X.shape}")
+    
+    # Scale features
+    print("Scaling features...")
+    scaler = StandardScaler()
+    X_scaled = scaler.fit_transform(X)
+    
+    # Build autoencoder
+    print("Building autoencoder...")
+    input_dim = X_scaled.shape[1]
+    
+    input_layer = Input(shape=(input_dim,))
+    encoded = Dense(64, activation='relu')(input_layer)
+    encoded = Dense(encoding_dim, activation='relu')(encoded)
+    decoded = Dense(64, activation='relu')(encoded)
+    decoded = Dense(input_dim, activation='linear')(decoded)
+    
+    autoencoder = Model(input_layer, decoded)
+    autoencoder.compile(optimizer='adam', loss='mse')
+    
+    print(f"Autoencoder architecture:")
+    autoencoder.summary()
+    
+    # Train
+    print("Training autoencoder...")
+    history = autoencoder.fit(
+        X_scaled, X_scaled,
+        epochs=epochs,
+        batch_size=batch_size,
+        validation_split=0.1,
+        verbose=1
+    )
+    
+    # Compute reconstruction errors
+    print("Computing reconstruction errors...")
+    X_pred = autoencoder.predict(X_scaled, verbose=0)
+    recon_errors = np.mean((X_scaled - X_pred) ** 2, axis=1)
+    
+    df['recon_error'] = recon_errors
+    
+    # Compute per-job thresholds (97th percentile)
+    df['anom_threshold_job'] = df.groupby('job_nm')['recon_error'].transform(
+        lambda s: np.percentile(s, 97) if len(s) > 0 else np.percentile(recon_errors, 97)
+    )
+    df['is_anomaly_job'] = (df['recon_error'] > df['anom_threshold_job']).astype(int)
+    
+    print(f"\nReconstruction error stats:")
+    print(f"  Mean: {recon_errors.mean():.6f}")
+    print(f"  Std: {recon_errors.std():.6f}")
+    print(f"  Min: {recon_errors.min():.6f}")
+    print(f"  Max: {recon_errors.max():.6f}")
+    print(f"  97th percentile: {np.percentile(recon_errors, 97):.6f}")
+    print(f"  Anomalies detected: {df['is_anomaly_job'].sum()} ({df['is_anomaly_job'].mean()*100:.2f}%)")
+    
+    # Save model and artifacts
+    model_path = os.path.join(output_dir, 'anomaly_autoencoder_cpu.keras')
+    autoencoder.save(model_path)
+    print(f"\nAutoencoder saved to {model_path}")
+    
+    scaler_path = os.path.join(output_dir, 'anomaly_scaler.joblib')
+    joblib.dump(scaler, scaler_path)
+    print(f"Scaler saved to {scaler_path}")
+    
+    feature_list_path = os.path.join(output_dir, 'anomaly_features.joblib')
+    joblib.dump(num_cols, feature_list_path)
+    print(f"Feature list saved to {feature_list_path}")
+    
+    # Save global threshold (97th percentile)
+    global_threshold = np.percentile(recon_errors, 97)
+    threshold_path = os.path.join(output_dir, 'anomaly_threshold.joblib')
+    joblib.dump(global_threshold, threshold_path)
+    print(f"Global threshold saved to {threshold_path} (value: {global_threshold:.6f})")
+    
+    return autoencoder, scaler, X_scaled, recon_errors
+
+
+if __name__ == '__main__':
+    import sys
+    import glob
+    
+    # Default CSV paths (can be overridden via command line)
+    if len(sys.argv) > 1:
+        # Handle glob patterns (works in both PowerShell and bash)
+        csv_paths = []
+        for arg in sys.argv[1:]:
+            # Expand glob patterns
+            expanded = glob.glob(arg)
+            if expanded:
+                csv_paths.extend(expanded)
+            else:
+                # If no match, use as-is (might be a specific file)
+                csv_paths.append(arg)
+    else:
+        # Default: all CSV files in data directory
+        csv_paths = glob.glob('data/*.csv')
+        if not csv_paths:
+            csv_paths = [
+                'data/true_export_report_20260120.csv',
+                'data/true_export_report_20260121.csv'
+            ]
+    
+    if not csv_paths:
+        print("Error: No CSV files found!")
+        print("Usage: python train_anomaly.py [file1.csv] [file2.csv] ...")
+        print("   or: python train_anomaly.py data/*.csv")
+        sys.exit(1)
+    
+    print(f"Training with {len(csv_paths)} file(s):")
+    for path in csv_paths:
+        print(f"  - {path}")
+    print()
+    
+    train_anomaly_detector(csv_paths)

train_job_failure.py

@@ -0,0 +1,257 @@
+"""
+Training script for Job Failure Prediction using XGBoost.
+"""
+import pandas as pd
+import numpy as np
+from sklearn.compose import ColumnTransformer
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler, OneHotEncoder
+from sklearn.impute import SimpleImputer
+from sklearn.metrics import (
+    classification_report, confusion_matrix, roc_auc_score,
+    precision_recall_fscore_support
+)
+from xgboost import XGBClassifier
+import joblib
+import os
+from preprocessing import engineer_features, get_feature_columns, save_feature_schema
+
+
+def train_job_failure_model(
+    train_csv_paths: list,
+    output_dir: str = 'models',
+    test_size: float = 0.2,
+    random_state: int = 42
+):
+    """
+    Train XGBoost model for job failure prediction.
+    
+    Args:
+        train_csv_paths: List of CSV file paths to load
+        output_dir: Directory to save model artifacts
+        test_size: Fraction of data to use for testing (time-based split)
+        random_state: Random seed
+    """
+    os.makedirs(output_dir, exist_ok=True)
+    
+    # Load and combine data
+    print("Loading data...")
+    dfs = []
+    for csv_path in train_csv_paths:
+        if os.path.exists(csv_path):
+            df = pd.read_csv(csv_path)
+            dfs.append(df)
+            print(f"  Loaded {len(df)} rows from {csv_path}")
+        else:
+            print(f"  Warning: {csv_path} not found, skipping")
+    
+    if not dfs:
+        raise ValueError("No data files found!")
+    
+    df = pd.concat(dfs, ignore_index=True)
+    print(f"Total rows: {len(df)}")
+    
+    # Engineer features
+    print("Engineering features...")
+    df = engineer_features(df)
+    
+    # Diagnostic: Check class distribution over time
+    if 'job_start_time' in df.columns and df['job_start_time'].notna().any():
+        df['job_start_time'] = pd.to_datetime(df['job_start_time'], errors='coerce')
+        df_sorted = df.sort_values('job_start_time')
+        print("\nClass distribution over time (first 10% vs last 10%):")
+        first_10pct = int(len(df_sorted) * 0.1)
+        last_10pct = int(len(df_sorted) * 0.9)
+        print(f"  First 10%: {df_sorted.iloc[:first_10pct]['is_failed'].value_counts().to_dict()}")
+        print(f"  Last 10%: {df_sorted.iloc[last_10pct:]['is_failed'].value_counts().to_dict()}")
+    
+    # Save feature schema
+    schema_path = os.path.join(output_dir, 'feature_schema.json')
+    save_feature_schema(schema_path)
+    
+    # Get feature columns
+    feature_cols = get_feature_columns()
+    num_cols = feature_cols['numeric']
+    cat_cols = feature_cols['categorical']
+    
+    # Prepare features and target
+    # Use current run failure as target (can be extended to future prediction)
+    df = df.sort_values(['job_nm', 'job_start_time']).reset_index(drop=True)
+    
+    # Ensure all feature columns exist
+    for col in num_cols + cat_cols:
+        if col not in df.columns:
+            if col in num_cols:
+                df[col] = 0.0
+            else:
+                df[col] = ''
+    
+    X = df[num_cols + cat_cols].copy()
+    y = df['is_failed'].copy()
+    
+    print(f"Features: {len(num_cols)} numeric, {len(cat_cols)} categorical")
+    print(f"Target distribution (full dataset): {y.value_counts().to_dict()}")
+    
+    # Diagnostic: Check status column values
+    if 'status' in df.columns:
+        print(f"Status column values: {df['status'].value_counts().to_dict()}")
+        print(f"Status column unique values: {df['status'].unique().tolist()}")
+    
+    # Check if we have both classes
+    unique_classes = y.unique()
+    if len(unique_classes) < 2:
+        raise ValueError(
+            f"Dataset contains only one class: {unique_classes}. "
+            "XGBoost requires at least 2 classes for binary classification. "
+            "Please check your data - all jobs may have the same status."
+        )
+    
+    # Time-based split (no shuffle)
+    cut_idx = int(len(df) * (1 - test_size))
+    X_train = X.iloc[:cut_idx]
+    X_test = X.iloc[cut_idx:]
+    y_train = y.iloc[:cut_idx]
+    y_test = y.iloc[cut_idx:]
+    
+    print(f"Train size: {len(X_train)}, Test size: {len(X_test)}")
+    print(f"Train target distribution: {y_train.value_counts().to_dict()}")
+    print(f"Test target distribution: {y_test.value_counts().to_dict()}")
+    
+    # Validate training set has both classes
+    train_unique_classes = y_train.unique()
+    if len(train_unique_classes) < 2:
+        print(f"\nWARNING: Training set contains only class(es): {train_unique_classes}")
+        print("This may be due to time-based split. Trying stratified approach...")
+        
+        # Try stratified split instead
+        from sklearn.model_selection import train_test_split
+        X_train, X_test, y_train, y_test = train_test_split(
+            X, y, 
+            test_size=test_size, 
+            stratify=y, 
+            random_state=random_state
+        )
+        print(f"Using stratified split instead:")
+        print(f"Train size: {len(X_train)}, Test size: {len(X_test)}")
+        print(f"Train target distribution: {y_train.value_counts().to_dict()}")
+        print(f"Test target distribution: {y_test.value_counts().to_dict()}")
+        
+        # Final check
+        if len(y_train.unique()) < 2:
+            raise ValueError(
+                f"Even with stratified split, training set has only one class: {y_train.unique()}. "
+                "This suggests severe class imbalance. Consider collecting more diverse data."
+            )
+    
+    # Build preprocessing pipeline
+    print("Building preprocessing pipeline...")
+    preprocess = ColumnTransformer([
+        ('num', Pipeline([
+            ('impute', SimpleImputer(strategy='constant', fill_value=0)),
+            ('scale', StandardScaler())
+        ]), num_cols),
+        ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), cat_cols)
+    ])
+    
+    # Build full pipeline
+    xgb = XGBClassifier(
+        tree_method='hist',
+        n_jobs=-1,
+        random_state=random_state,
+        eval_metric='logloss'
+    )
+    
+    pipe = Pipeline([
+        ('preprocess', preprocess),
+        ('model', xgb)
+    ])
+    
+    # Train
+    print("Training model...")
+    pipe.fit(X_train, y_train)
+    
+    # Evaluate
+    print("Evaluating model...")
+    try:
+        y_pred = pipe.predict(X_test)
+        y_proba = pipe.predict_proba(X_test)[:, 1]
+    except AttributeError as e:
+        if '__sklearn_tags__' in str(e):
+            # Workaround for XGBoost/scikit-learn compatibility issue
+            print("Warning: Compatibility issue detected. Using direct model prediction...")
+            # Get the preprocessed test data
+            X_test_processed = pipe.named_steps['preprocess'].transform(X_test)
+            # Use the model directly
+            xgb_model = pipe.named_steps['model']
+            y_pred = xgb_model.predict(X_test_processed)
+            y_proba = xgb_model.predict_proba(X_test_processed)[:, 1]
+        else:
+            raise
+    
+    print("\n=== Classification Report ===")
+    print(classification_report(y_test, y_pred))
+    
+    print("\n=== Confusion Matrix ===")
+    print(confusion_matrix(y_test, y_pred))
+    
+    if len(np.unique(y_test)) > 1:
+        auc = roc_auc_score(y_test, y_proba)
+        print(f"\nAUC-ROC: {auc:.4f}")
+    
+    precision, recall, f1, _ = precision_recall_fscore_support(
+        y_test, y_pred, average='binary', zero_division=0
+    )
+    print(f"Precision: {precision:.4f}, Recall: {recall:.4f}, F1: {f1:.4f}")
+    
+    # Save model
+    model_path = os.path.join(output_dir, 'job_fail_pipeline_cpu.joblib')
+    joblib.dump(pipe, model_path)
+    print(f"\nModel saved to {model_path}")
+    
+    # Save background sample for SHAP (small sample for efficiency)
+    X_train_sample = X_train.sample(min(100, len(X_train)), random_state=random_state)
+    X_train_encoded = preprocess.transform(X_train_sample)
+    background_path = os.path.join(output_dir, 'shap_background.npy')
+    np.save(background_path, X_train_encoded)
+    print(f"SHAP background sample saved to {background_path}")
+    
+    return pipe, X_test, y_test
+
+
+if __name__ == '__main__':
+    import sys
+    import glob
+    
+    # Default CSV paths (can be overridden via command line)
+    if len(sys.argv) > 1:
+        # Handle glob patterns (works in both PowerShell and bash)
+        csv_paths = []
+        for arg in sys.argv[1:]:
+            # Expand glob patterns
+            expanded = glob.glob(arg)
+            if expanded:
+                csv_paths.extend(expanded)
+            else:
+                # If no match, use as-is (might be a specific file)
+                csv_paths.append(arg)
+    else:
+        # Default: all CSV files in data directory
+        csv_paths = glob.glob('data/*.csv')
+        if not csv_paths:
+            csv_paths = [
+                'data/true_export_report_20260120.csv',
+                'data/true_export_report_20260121.csv'
+            ]
+    
+    if not csv_paths:
+        print("Error: No CSV files found!")
+        print("Usage: python train_job_failure.py [file1.csv] [file2.csv] ...")
+        print("   or: python train_job_failure.py data/*.csv")
+        sys.exit(1)
+    
+    print(f"Training with {len(csv_paths)} file(s):")
+    for path in csv_paths:
+        print(f"  - {path}")
+    print()
+    
+    train_job_failure_model(csv_paths)