Deploy to Hugging Face Spaces: Add application files and dependencies

- Add FastAPI application (app.py)
- Add preprocessing module (preprocessing.py)
- Add model utilities (model_utils.py)
- Add Dockerfile for HF Spaces deployment
- Add requirements.txt
- Add deployment documentation
- Add configuration files

.dockerignore

@@ -0,0 +1,50 @@
+# Python
+__pycache__
+*.pyc
+*.pyo
+*.pyd
+.Python
+*.so
+*.egg
+*.egg-info
+dist
+build
+.venv
+venv/
+ENV/
+
+# Git
+.git
+.gitignore
+.gitattributes
+
+# Environment
+.env
+*.log
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Documentation (keep README.md for HF Spaces)
+*.md
+!README.md
+
+# Training data (not needed in production)
+data/
+*.csv
+
+# Development files
+.ipynb_checkpoints
+.pytest_cache
+.coverage
+htmlcov/
+test_*.py
+train_*.py
+
+# Docker
+docker-compose.yml
+.dockerignore
+
+# Keep models/ directory - models are committed to repo
+# Keep all application code (app.py, model_utils.py, preprocessing.py)

.gitignore

@@ -0,0 +1,65 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+venv/
+ENV/
+env/
+.venv
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# Models and data
+models/
+data/
+*.joblib
+*.keras
+*.h5
+*.pkl
+*.npy
+
+# Logs
+*.log
+logs/
+
+# Environment variables
+.env
+.env.local
+
+# Jupyter
+.ipynb_checkpoints/
+*.ipynb
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+
+# OS
+.DS_Store
+Thumbs.db

DEPLOYMENT.md

@@ -0,0 +1,410 @@
+# Hugging Face Spaces Deployment Guide
+
+คู่มือการ deploy API ไปยัง Hugging Face Spaces สำหรับ production
+
+## 📋 คำถามสำคัญก่อน Deploy
+
+### Q1: ใช้ Docker Space หรือ Python Space?
+**คำตอบ: Docker Space ✅**
+
+**เหตุผล:**
+- มี TensorFlow + XGBoost + joblib หลาย dependencies
+- ต้องควบคุมเวอร์ชัน dependencies เอง
+- Python Space อาจมี dependency conflicts
+- Docker Space ให้ความยืดหยุ่นและควบคุมได้มากกว่า
+
+### Q2: โหลด model จากไหน?
+**คำตอบ: Commit models/ เข้า repo ✅**
+
+**เหตุผล:**
+- ✅ Predictable - models อยู่ใน repo เดียวกับ code
+- ✅ ไม่ต้อง auth - ไม่ต้องใช้ HF token
+- ✅ Startup เร็ว - ไม่ต้อง download ตอน startup
+- ✅ Rollback ง่าย - git revert ได้เลย
+
+**หมายเหตุ:** ถ้า models ใหญ่เกิน 1GB ให้พิจารณาใช้ HF Model Hub แทน
+
+### Q3: API Pattern?
+**คำตอบ: ตรงตาม microservice standard ✅**
+
+- `/health` - Health check
+- `/predict/job-fail` - Job failure prediction
+- `/detect/anomaly` - Anomaly detection
+
+---
+
+## 🚀 Step-by-Step Deployment
+
+### Step 1: เตรียม Models
+
+1. **Train models** (ถ้ายังไม่มี):
+   ```bash
+   python train_job_failure.py data/*.csv
+   python train_anomaly.py data/*.csv
+   ```
+
+2. **ตรวจสอบ models directory:**
+   ```bash
+   ls -lh models/
+   ```
+   
+   ต้องมีไฟล์เหล่านี้:
+   - `job_fail_pipeline_cpu.joblib`
+   - `anomaly_autoencoder_cpu.keras`
+   - `anomaly_scaler.joblib`
+   - `anomaly_features.joblib`
+   - `anomaly_threshold.joblib`
+   - `feature_schema.json`
+   - `shap_background.npy`
+
+3. **ตรวจสอบขนาดไฟล์:**
+   ```bash
+   du -sh models/
+   ```
+   
+   ⚠️ **สำคัญ:** Hugging Face Spaces จำกัด repo size ~50GB (free tier)
+   - ถ้า models > 1GB ให้พิจารณาใช้ Git LFS หรือ HF Model Hub
+
+### Step 2: เตรียม Code สำหรับ Deploy
+
+1. **ตรวจสอบไฟล์ที่จำเป็น:**
+   ```bash
+   # ไฟล์ที่ต้องมี
+   - Dockerfile
+   - requirements.txt
+   - app.py
+   - model_utils.py
+   - preprocessing.py
+   - models/ (directory with all model files)
+   - README.md หรือ README_HF_SPACE.md
+   ```
+
+2. **ตรวจสอบ .dockerignore:**
+   - ต้อง exclude `data/`, `*.csv`, `train_*.py`, `test_*.py`
+   - ต้อง **include** `models/` directory
+
+3. **ตรวจสอบ Dockerfile:**
+   - Port ต้องเป็น `7860` (HF Spaces default)
+   - Health check ต้องใช้ port `7860`
+
+### Step 3: สร้าง Hugging Face Space
+
+1. **ไปที่ Hugging Face Spaces:**
+   - เปิด https://huggingface.co/spaces
+   - คลิก "Create new Space"
+
+2. **ตั้งค่า Space:**
+   - **Space name:** `job-failure-prediction-api` (หรือชื่อที่ต้องการ)
+   - **SDK:** เลือก **Docker** (ไม่ใช่ Python)
+   - **Visibility:** Public หรือ Private (ตามต้องการ)
+   - **Hardware:** CPU Basic (free) หรือ GPU (ถ้าต้องการ)
+
+3. **คลิก "Create Space"**
+
+### Step 4: Upload Code ไปยัง Space
+
+#### Option A: ใช้ Git (แนะนำ)
+
+1. **Clone Space repository:**
+   ```bash
+   git clone https://huggingface.co/spaces/[your-username]/[space-name]
+   cd [space-name]
+   ```
+
+2. **Copy files จาก project:**
+   ```bash
+   # Copy application files
+   cp ../ml_service/app.py .
+   cp ../ml_service/model_utils.py .
+   cp ../ml_service/preprocessing.py .
+   cp ../ml_service/requirements.txt .
+   cp ../ml_service/Dockerfile .
+   
+   # Copy models directory
+   cp -r ../ml_service/models .
+   
+   # Copy README for HF Space
+   cp ../ml_service/README_HF_SPACE.md README.md
+   ```
+
+3. **Commit และ Push:**
+   ```bash
+   git add .
+   git commit -m "Initial deployment: Job Failure Prediction API"
+   git push
+   ```
+
+#### Option B: ใช้ Web UI
+
+1. **Upload files ผ่าน web interface:**
+   - ไปที่ Space page
+   - คลิก "Files and versions" tab
+   - Upload files ทีละไฟล์:
+     - `Dockerfile`
+     - `requirements.txt`
+     - `app.py`
+     - `model_utils.py`
+     - `preprocessing.py`
+     - `README.md` (copy จาก README_HF_SPACE.md)
+
+2. **Upload models:**
+   - สร้าง folder `models/`
+   - Upload model files ทั้งหมดเข้าไป
+
+⚠️ **หมายเหตุ:** Git method เร็วกว่า��ละเหมาะกับ large files มากกว่า
+
+### Step 5: รอ Build และ Deploy
+
+1. **ตรวจสอบ Build Logs:**
+   - ไปที่ Space page
+   - คลิก "Logs" tab
+   - ดู build progress
+
+2. **Build time:** ประมาณ 5-15 นาที (ขึ้นกับขนาด models)
+
+3. **ตรวจสอบ Health:**
+   - เมื่อ build เสร็จ ไปที่ `https://[space-name].hf.space/health`
+   - ควรได้ response:
+     ```json
+     {
+       "status": "healthy",
+       "service": "job-failure-prediction",
+       "models_loaded": {
+         "predictor": true,
+         "anomaly_detector": true
+       }
+     }
+     ```
+
+### Step 6: ทดสอบ API
+
+1. **Test Health Endpoint:**
+   ```bash
+   curl https://[your-username]-[space-name].hf.space/health
+   ```
+
+2. **Test Prediction Endpoint:**
+   ```bash
+   curl -X POST https://[your-username]-[space-name].hf.space/predict/job-fail \
+     -H "Content-Type: application/json" \
+     -d '{
+       "zone": "prod",
+       "job_nm": "test_job",
+       "job_start_time": "2026-01-21T01:00:00",
+       "duration_sec": 3600,
+       "status": "SUCCESS"
+     }'
+   ```
+
+3. **Test Anomaly Detection:**
+   ```bash
+   curl -X POST https://[your-username]-[space-name].hf.space/detect/anomaly \
+     -H "Content-Type: application/json" \
+     -d '{
+       "features": {
+         "duration_sec": 5400,
+         "duration_zscore": 1.6,
+         "err_msg_len": 0
+       }
+     }'
+   ```
+
+4. **Test API Documentation:**
+   - เปิด `https://[space-name].hf.space/docs` (Swagger UI)
+   - เปิด `https://[space-name].hf.space/redoc` (ReDoc)
+
+### Step 7: Configuration (Optional)
+
+#### Environment Variables
+
+ถ้าต้องการ config แบบ dynamic สามารถเพิ่ม environment variables ใน Space settings:
+
+1. ไปที่ Space → Settings → Variables
+2. เพิ่ม variables (ถ้าต้องการ):
+   - `MODEL_PATH` (default: `models/`)
+   - `LOG_LEVEL` (default: `INFO`)
+
+#### Hardware Upgrade
+
+ถ้าต้องการ performance มากขึ้น:
+1. ไปที่ Settings → Hardware
+2. เลือก:
+   - **CPU Basic** (free) - เหมาะกับ production แรก
+   - **CPU Upgrade** (paid) - สำหรับ traffic สูง
+   - **GPU** (paid) - ถ้า models ใหญ่และต้องการ inference เร็ว
+
+---
+
+## 🔍 Troubleshooting
+
+### Problem: Build Fails
+
+**สาเหตุที่เป็นไปได้:**
+1. Dockerfile syntax error
+2. Dependencies conflict
+3. Models ไม่มีใน repo
+
+**แก้ไข:**
+```bash
+# ตรวจสอบ Dockerfile
+docker build -t test-build .
+
+# ตรวจสอบ logs ใน HF Spaces
+# ดู error message ใน Logs tab
+```
+
+### Problem: Models Not Loading
+
+**สาเหตุ:**
+- Models directory ไม่ถูก copy
+- Path ไม่ถูกต้อง
+
+**แก้ไข:**
+```bash
+# ตรวจสอบใน Space
+# ไปที่ Files tab → ดูว่ามี models/ directory หรือไม่
+
+# ตรวจสอบ logs
+# ดู error message เกี่ยวกับ model loading
+```
+
+### Problem: Port Error
+
+**สาเหตุ:**
+- App ใช้ port ไม่ใช่ 7860
+
+**แก้ไข:**
+- ตรวจสอบ `app.py` และ `Dockerfile` ว่าใช้ port 7860
+
+### Problem: Slow Startup
+
+**สาเหตุ:**
+- Models ใหญ่เกินไป
+- Loading models จาก network
+
+**แก้ไข:**
+- ใช้ models ที่ commit ใน repo (ไม่ download)
+- ตรวจสอบ model size และ optimize ถ้าจำเป็น
+
+---
+
+## 📊 Monitoring
+
+### Health Check
+
+ตั้งค่า monitoring เพื่อตรวจสอบ health:
+```bash
+# Cron job หรือ monitoring service
+curl https://[space-name].hf.space/health
+```
+
+### Logs
+
+ดู logs ใน Hugging Face Spaces:
+1. ไปที่ Space → Logs tab
+2. ดู application logs และ errors
+
+### Metrics
+
+Hugging Face Spaces แสดง:
+- Request count
+- Response time
+- Error rate
+- Resource usage
+
+---
+
+## 🔄 Updates และ Rollback
+
+### Update Models
+
+1. **Train new models:**
+   ```bash
+   python train_job_failure.py data/*.csv
+   python train_anomaly.py data/*.csv
+   ```
+
+2. **Update in Space:**
+   ```bash
+   cd [space-repo]
+   cp -r ../ml_service/models .
+   git add models/
+   git commit -m "Update models"
+   git push
+   ```
+
+3. **Space จะ rebuild อัตโนมัติ**
+
+### Rollback
+
+```bash
+# ดู commit history
+git log
+
+# Rollback to previous version
+git revert HEAD
+git push
+```
+
+---
+
+## 🎯 Best Practices
+
+1. **Version Control:**
+   - ใช้ Git tags สำหรับ releases
+   - Commit messages ที่ชัดเจน
+
+2. **Testing:**
+   - Test ทุก endpoint หลัง deploy
+   - ใช้ staging space สำหรับ testing
+
+3. **Documentation:**
+   - อัพเดท README.md เมื่อมีการเปลี่ยนแปลง
+   - Document API changes
+
+4. **Monitoring:**
+   - ตั้งค่า alerts สำหรับ health check failures
+   - Monitor response times
+
+5. **Security:**
+   - ใช้ Private Space สำหรับ sensitive data
+   - ตรวจสอบ CORS settings
+
+---
+
+## 📚 Resources
+
+- [Hugging Face Spaces Documentation](https://huggingface.co/docs/hub/spaces)
+- [Docker Spaces Guide](https://huggingface.co/docs/hub/spaces-sdks-docker)
+- [FastAPI Documentation](https://fastapi.tiangolo.com/)
+
+---
+
+## ✅ Checklist
+
+ก่อน deploy ตรวจสอบ:
+
+- [ ] Models ถูก train และอยู่ใน `models/` directory
+- [ ] Dockerfile ใช้ port 7860
+- [ ] Health check ใช้ port 7860
+- [ ] `.dockerignore` exclude files ที่ไม่จำเป็น
+- [ ] `requirements.txt` มี dependencies ครบ
+- [ ] `README.md` สำหรับ HF Space พร้อม
+- [ ] Test API locally ก่อน deploy
+- [ ] Space ถูกสร้างและตั้งค่าเป็น Docker
+- [ ] Code และ models ถูก upload ครบ
+- [ ] Build สำเร็จและ health check ผ่าน
+- [ ] ทุก endpoint ทำงานถูกต้อง
+
+---
+
+## 🎉 Success!
+
+เมื่อ deploy สำเร็จ คุณจะมี production-ready API ที่:
+- ✅ Auto-scaling
+- ✅ HTTPS enabled
+- ✅ Global CDN
+- ✅ Monitoring และ logging
+- ✅ Easy rollback
+
+API URL: `https://[your-username]-[space-name].hf.space`

DEPLOYMENT_SUMMARY.md

@@ -0,0 +1,180 @@
+# 🚀 Hugging Face Deployment - สรุปการเตรียมพร้อม
+
+## ✅ คำตอบ 3 คำถามสำคัญ
+
+### Q1: ใช้ Docker Space หรือ Python Space?
+**คำตอบ: Docker Space ✅**
+
+**เหตุผล:**
+- มี TensorFlow + XGBoost + joblib หลาย dependencies
+- ต้องควบคุมเวอร์ชัน dependencies เอง
+- Python Space อาจมี dependency conflicts
+- Docker Space ให้ความยืดหยุ่นและควบคุมได้มากกว่า
+
+### Q2: โหลด model จากไหน?
+**คำตอบ: Commit models/ เข้า repo ✅**
+
+**เหตุผล:**
+- ✅ Predictable - models อยู่ใน repo เดียวกับ code
+- ✅ ไม่ต้อง auth - ไม่ต้องใช้ HF token
+- ✅ Startup เร็ว - ไม่ต้อง download ตอน startup
+- ✅ Rollback ง่าย - git revert ได้เลย
+
+### Q3: API Pattern?
+**คำตอบ: ตรงตาม microservice standard ✅**
+
+- `/health` - Health check
+- `/predict/job-fail` - Job failure prediction
+- `/detect/anomaly` - Anomaly detection
+
+---
+
+## 📝 สิ่งที่ทำเสร็จแล้ว
+
+### 1. ✅ Dockerfile (อัพเดทแล้ว)
+- ใช้ port **7860** (Hugging Face Spaces default)
+- Health check ใช้ port 7860
+- Optimize สำหรับ production
+- เพิ่ม build dependencies
+
+### 2. ✅ .dockerignore (อัพเดทแล้ว)
+- Exclude files ที่ไม่จำเป็น (data/, train_*.py, test_*.py)
+- **Keep** models/ directory (models จะถูก commit)
+- Optimize build time
+
+### 3. ✅ model_utils.py (เพิ่ม Environment Variables)
+- รองรับ `MODEL_BASE_PATH` environment variable
+- รองรับ individual model paths ผ่าน env vars
+- Default ยังคงเป็น local models/ directory
+- พร้อมสำหรับ future: load จาก HF Hub (ถ้าต้องการ)
+
+### 4. ✅ README_HF_SPACE.md (สร้างใหม่)
+- README สำหรับ Hugging Face Space
+- API documentation
+- Example usage
+- Configuration guide
+
+### 5. ✅ DEPLOYMENT.md (สร้างใหม่)
+- Step-by-step deployment guide
+- Troubleshooting
+- Best practices
+- Checklist
+
+### 6. ✅ app.py (ตรวจสอบแล้ว)
+- ใช้ port 7860 ✅
+- FastAPI setup ถูกต้อง ✅
+- CORS enabled ✅
+
+---
+
+## 🎯 ขั้นตอนต่อไป (Deploy)
+
+### Step 1: เตรียม Models
+```bash
+# ตรวจสอบว่า models ถูก train แล้ว
+ls -lh models/
+
+# ต้องมีไฟล์เหล่านี้:
+# - job_fail_pipeline_cpu.joblib
+# - anomaly_autoencoder_cpu.keras
+# - anomaly_scaler.joblib
+# - anomaly_features.joblib
+# - anomaly_threshold.joblib
+# - feature_schema.json
+# - shap_background.npy
+```
+
+### Step 2: สร้าง Hugging Face Space
+1. ไปที่ https://huggingface.co/spaces
+2. คลิก "Create new Space"
+3. ตั้งค่า:
+   - **SDK:** Docker (ไม่ใช่ Python!)
+   - **Hardware:** CPU Basic (free) หรือ upgrade
+   - **Visibility:** Public/Private
+
+### Step 3: Upload Code
+```bash
+# Option A: Git (แนะนำ)
+git clone https://huggingface.co/spaces/[username]/[space-name]
+cd [space-name]
+
+# Copy files
+cp ../ml_service/app.py .
+cp ../ml_service/model_utils.py .
+cp ../ml_service/preprocessing.py .
+cp ../ml_service/requirements.txt .
+cp ../ml_service/Dockerfile .
+cp ../ml_service/README_HF_SPACE.md README.md
+cp -r ../ml_service/models .
+
+# Commit
+git add .
+git commit -m "Initial deployment"
+git push
+```
+
+### Step 4: รอ Build
+- ดู logs ใน Space → Logs tab
+- Build time: ~5-15 นาที
+
+### Step 5: Test
+```bash
+# Health check
+curl https://[space-name].hf.space/health
+
+# Test prediction
+curl -X POST https://[space-name].hf.space/predict/job-fail \
+  -H "Content-Type: application/json" \
+  -d '{"zone": "prod", "job_nm": "test", "job_start_time": "2026-01-21T01:00:00", "duration_sec": 3600}'
+```
+
+---
+
+## 📋 Checklist ก่อน Deploy
+
+- [x] Dockerfile ใช้ port 7860
+- [x] Health check ใช้ port 7860
+- [x] .dockerignore exclude files ที่ไม่จำเป็น
+- [x] models/ directory พร้อม commit
+- [x] Environment variables support
+- [x] README_HF_SPACE.md พร้อม
+- [x] DEPLOYMENT.md มีคำแนะนำครบ
+- [ ] Models ถูก train และอยู่ใน models/
+- [ ] Test API locally ก่อน deploy
+- [ ] สร้าง Hugging Face Space
+- [ ] Upload code และ models
+- [ ] Test ทุก endpoint หลัง deploy
+
+---
+
+## 🔗 ไฟล์ที่สำคัญ
+
+1. **Dockerfile** - Docker configuration สำหรับ HF Spaces
+2. **README_HF_SPACE.md** - README สำหรับ Space page
+3. **DEPLOYMENT.md** - คู่มือ deploy แบบละเอียด
+4. **.dockerignore** - Exclude files สำหรับ Docker build
+5. **model_utils.py** - รองรับ environment variables
+
+---
+
+## 🎉 พร้อม Deploy!
+
+ทุกอย่างพร้อมแล้วสำหรับ production deployment ไปยัง Hugging Face Spaces
+
+**API URL หลัง deploy:**
+```
+https://[your-username]-[space-name].hf.space
+```
+
+**Documentation:**
+- Swagger UI: `/docs`
+- ReDoc: `/redoc`
+- Health: `/health`
+
+---
+
+## 📚 อ่านเพิ่มเติม
+
+- [DEPLOYMENT.md](DEPLOYMENT.md) - คู่มือ deploy แบบละเอียด
+- [README_HF_SPACE.md](README_HF_SPACE.md) - README สำหรับ Space
+- [README.md](README.md) - Documentation หลัก

Dockerfile

@@ -20,6 +20,13 @@ RUN pip install --no-cache-dir --upgrade pip && \
 # Copy application code and models
 COPY --chown=user . /app
 
+# Verify critical files exist (helps debug if files are missing)
+RUN ls -la /app/*.py && \
+    test -f /app/app.py || (echo "ERROR: app.py not found!" && exit 1) && \
+    test -f /app/preprocessing.py || (echo "ERROR: preprocessing.py not found!" && exit 1) && \
+    test -f /app/model_utils.py || (echo "ERROR: model_utils.py not found!" && exit 1) && \
+    echo "All required Python files found!"
+
 # Ensure models directory exists (models should be committed to repo)
 RUN mkdir -p /app/models /app/data
 

README.md

@@ -1,10 +1,286 @@
----
-title: Ml Service
-emoji: 🐨
-colorFrom: purple
-colorTo: blue
-sdk: docker
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Job Failure Prediction & Anomaly Detection ML Service
+
+Production-ready ML service for predicting job failures and detecting anomalies in job execution data.
+
+## Features
+
+- **Job Failure Prediction**: XGBoost-based classifier with SHAP explainability
+- **Anomaly Detection**: Autoencoder-based unsupervised anomaly detection
+- **FastAPI REST API**: Production-ready endpoints
+- **Docker Support**: Containerized deployment
+- **n8n Integration**: Ready-to-use workflow examples
+
+## Quick Start
+
+### 1. Training Models
+
+Place your CSV files in the `data/` directory:
+
+```bash
+mkdir -p data models
+# Copy your CSV files: true_export_report_20260120.csv, true_export_report_20260121.csv
+```
+
+Train the job failure prediction model:
+
+```bash
+python train_job_failure.py data/true_export_report_20260120.csv data/true_export_report_20260121.csv
+```
+
+Train the anomaly detection model:
+
+```bash
+python train_anomaly.py data/true_export_report_20260120.csv data/true_export_report_20260121.csv
+```
+
+Models will be saved to `models/` directory.
+
+### 2. Running the Service
+
+#### Local Development
+
+```bash
+pip install -r requirements.txt
+uvicorn app:app --host 0.0.0.0 --port 8000
+```
+
+#### Docker
+
+```bash
+docker-compose up --build
+```
+
+The service will be available at `http://localhost:8000`
+
+### 3. API Documentation
+
+Once running, visit:
+- Swagger UI: `http://localhost:8000/docs`
+- ReDoc: `http://localhost:8000/redoc`
+
+## API Endpoints
+
+### Health Check
+
+```bash
+GET /health
+```
+
+### Job Failure Prediction
+
+```bash
+POST /predict/job-fail
+Content-Type: application/json
+
+{
+  "zone": "prod",
+  "job_nm": "daily_export_customer",
+  "job_start_time": "2026-01-21T01:00:00",
+  "duration_sec": 5400,
+  "status": "SUCCESS",
+  "err_msg": "",
+  "explain": true
+}
+```
+
+**Response:**
+```json
+{
+  "fail_probability": 0.79,
+  "risk_level": "MEDIUM",
+  "top_drivers": [
+    {
+      "feature": "failure_rate_7",
+      "shap_value": 0.30,
+      "effect": "increase"
+    }
+  ],
+  "recommended_actions": [
+    "Monitor upstream dependencies and recent job history"
+  ]
+}
+```
+
+### Anomaly Detection
+
+```bash
+POST /detect/anomaly
+Content-Type: application/json
+
+{
+  "features": {
+    "duration_sec": 5400,
+    "duration_zscore": 1.6,
+    "err_msg_len": 0
+  },
+  "threshold": 0.01
+}
+```
+
+**Response:**
+```json
+{
+  "reconstruction_error": 0.0235,
+  "is_anomaly": true,
+  "threshold": 0.01,
+  "top_drivers": [
+    {
+      "feature": "duration_zscore",
+      "error": 0.0142
+    }
+  ]
+}
+```
+
+## Project Structure
+
+```
+ml_service/
+├── app.py                      # FastAPI application
+├── preprocessing.py            # Data preprocessing and feature engineering
+├── model_utils.py             # Model inference with SHAP
+├── train_job_failure.py       # Training script for failure prediction
+├── train_anomaly.py           # Training script for anomaly detection
+├── requirements.txt           # Python dependencies
+├── Dockerfile                 # Docker image definition
+├── docker-compose.yml         # Docker Compose configuration
+├── n8n_workflow_examples.md   # n8n integration guide
+├── n8n_workflow_job_monitoring.json  # Importable n8n workflow
+├── models/                    # Trained models (created after training)
+│   ├── job_fail_pipeline_cpu.joblib
+│   ├── anomaly_autoencoder_cpu.keras
+│   ├── anomaly_scaler.joblib
+│   ├── feature_schema.json
+│   └── ...
+└── data/                      # Training data (user-provided)
+    ├── true_export_report_20260120.csv
+    └── true_export_report_20260121.csv
+```
+
+## Features Engineering
+
+The service automatically engineers the following features:
+
+### Numeric Features
+- `duration_sec`: Job duration in seconds
+- `duration_zscore`: Z-score relative to rolling average
+- `avg_duration_7`: 7-day rolling average duration
+- `failure_rate_7`: 7-day rolling failure rate
+- `err_msg_len`: Error message length
+- `hour_sin`, `hour_cos`: Cyclical hour encoding
+
+### Categorical Features
+- `job_nm`: Job name
+- `tasksgroup_nm`: Task group name
+- `zone`: Environment/cluster
+- `is_zeppelin`: Zeppelin flag
+- `is_weekend`: Weekend indicator
+
+## Model Details
+
+### Job Failure Prediction
+- **Algorithm**: XGBoost Classifier
+- **Preprocessing**: StandardScaler for numeric, OneHotEncoder for categorical
+- **Explainability**: SHAP values for feature importance
+- **Output**: Failure probability (0-1), risk level, top drivers, recommended actions
+
+### Anomaly Detection
+- **Algorithm**: Autoencoder (TensorFlow/Keras)
+- **Architecture**: Input → 64 → 32 → 64 → Output
+- **Threshold**: Per-job 97th percentile or global threshold
+- **Output**: Reconstruction error, anomaly flag, top contributing features
+
+## n8n Integration
+
+See `n8n_workflow_examples.md` for detailed integration examples and `n8n_workflow_job_monitoring.json` for an importable workflow.
+
+### Quick n8n Setup
+
+1. Import `n8n_workflow_job_monitoring.json` into n8n
+2. Update the HTTP Request URLs to match your service endpoint
+3. Configure Slack credentials (or replace with your alerting system)
+4. Activate the workflow
+
+## Risk Levels
+
+- **MINIMAL**: `fail_probability < 0.3`
+- **LOW**: `0.3 <= fail_probability < 0.5`
+- **MEDIUM**: `0.5 <= fail_probability < 0.8`
+- **CRITICAL**: `fail_probability >= 0.8`
+
+## Alert Conditions
+
+### WARNING
+- `fail_probability >= 0.5 && fail_probability < 0.8`
+- OR `is_anomaly === true` with moderate reconstruction error
+
+### CRITICAL
+- `fail_probability >= 0.8`
+- OR `is_anomaly === true` with high reconstruction error (> 3x threshold)
+
+## Development
+
+### Running Tests
+
+```bash
+# Test prediction endpoint
+curl -X POST http://localhost:8000/predict/job-fail \
+  -H "Content-Type: application/json" \
+  -d '{
+    "zone": "prod",
+    "job_nm": "test_job",
+    "job_start_time": "2026-01-21T01:00:00",
+    "duration_sec": 3600,
+    "status": "SUCCESS"
+  }'
+```
+
+### Model Retraining
+
+Models should be retrained periodically as new data becomes available:
+
+```bash
+# Add new CSV files to data/ directory
+python train_job_failure.py data/*.csv
+python train_anomaly.py data/*.csv
+
+# Restart service to load new models
+docker-compose restart
+```
+
+## Monitoring
+
+The service includes a health check endpoint that reports model loading status:
+
+```bash
+GET /health
+```
+
+Response:
+```json
+{
+  "status": "healthy",
+  "service": "job-failure-prediction",
+  "models_loaded": {
+    "predictor": true,
+    "anomaly_detector": true
+  }
+}
+```
+
+## Production Deployment
+
+1. **Train models** on historical data
+2. **Save models** to `models/` directory
+3. **Build Docker image**: `docker build -t job-ml-service .`
+4. **Deploy** using docker-compose or Kubernetes
+5. **Configure n8n** workflows for monitoring
+6. **Set up alerting** (Slack, PagerDuty, etc.)
+
+## License
+
+[Your License Here]
+
+## Support
+
+For issues or questions, please refer to the documentation or contact the development team.

README_HF_SPACE.md

@@ -0,0 +1,179 @@
+---
+title: Job Failure Prediction & Anomaly Detection API
+emoji: 🔮
+colorFrom: blue
+colorTo: purple
+sdk: docker
+sdk_version: latest
+app_port: 7860
+pinned: false
+license: apache-2.0
+---
+
+# Job Failure Prediction & Anomaly Detection API
+
+Production-ready ML service for predicting job failures and detecting anomalies in job execution data.
+
+## 🚀 Quick Start
+
+The API is automatically deployed and available at:
+```
+https://[your-space-name].hf.space
+```
+
+## 📡 API Endpoints
+
+### Health Check
+```bash
+GET /health
+```
+
+### Job Failure Prediction
+```bash
+POST /predict/job-fail
+Content-Type: application/json
+
+{
+  "zone": "prod",
+  "job_nm": "daily_export_customer",
+  "job_start_time": "2026-01-21T01:00:00",
+  "duration_sec": 5400,
+  "status": "SUCCESS",
+  "err_msg": "",
+  "explain": true
+}
+```
+
+**Response:**
+```json
+{
+  "fail_probability": 0.79,
+  "risk_level": "MEDIUM",
+  "top_drivers": [
+    {
+      "feature": "failure_rate_7",
+      "shap_value": 0.30,
+      "effect": "increase"
+    }
+  ],
+  "recommended_actions": [
+    "Monitor upstream dependencies and recent job history"
+  ]
+}
+```
+
+### Anomaly Detection
+```bash
+POST /detect/anomaly
+Content-Type: application/json
+
+{
+  "features": {
+    "duration_sec": 5400,
+    "duration_zscore": 1.6,
+    "err_msg_len": 0
+  },
+  "threshold": 0.01
+}
+```
+
+**Response:**
+```json
+{
+  "reconstruction_error": 0.0235,
+  "is_anomaly": true,
+  "threshold": 0.01,
+  "top_drivers": [
+    {
+      "feature": "duration_zscore",
+      "error": 0.0142
+    }
+  ]
+}
+```
+
+## 📚 API Documentation
+
+Interactive API documentation is available at:
+- Swagger UI: `/docs`
+- ReDoc: `/redoc`
+
+## 🎯 Features
+
+- **Job Failure Prediction**: XGBoost-based classifier with SHAP explainability
+- **Anomaly Detection**: Autoencoder-based unsupervised anomaly detection
+- **FastAPI REST API**: Production-ready endpoints
+- **Docker Deployment**: Containerized for Hugging Face Spaces
+
+## 📊 Risk Levels
+
+- **MINIMAL**: `fail_probability < 0.3`
+- **LOW**: `0.3 <= fail_probability < 0.5`
+- **MEDIUM**: `0.5 <= fail_probability < 0.8`
+- **CRITICAL**: `fail_probability >= 0.8`
+
+## 🔧 Model Details
+
+### Job Failure Prediction
+- **Algorithm**: XGBoost Classifier
+- **Preprocessing**: StandardScaler for numeric, OneHotEncoder for categorical
+- **Explainability**: SHAP values for feature importance
+
+### Anomaly Detection
+- **Algorithm**: Autoencoder (TensorFlow/Keras)
+- **Architecture**: Input → 64 → 32 → 64 → Output
+- **Threshold**: Per-job 97th percentile or global threshold
+
+## 📝 Example Usage
+
+### Python
+```python
+import requests
+
+url = "https://[your-space-name].hf.space/predict/job-fail"
+response = requests.post(url, json={
+    "zone": "prod",
+    "job_nm": "daily_export",
+    "job_start_time": "2026-01-21T01:00:00",
+    "duration_sec": 5400,
+    "status": "SUCCESS",
+    "explain": True
+})
+print(response.json())
+```
+
+### cURL
+```bash
+curl -X POST https://[your-space-name].hf.space/predict/job-fail \
+  -H "Content-Type: application/json" \
+  -d '{
+    "zone": "prod",
+    "job_nm": "daily_export",
+    "job_start_time": "2026-01-21T01:00:00",
+    "duration_sec": 5400,
+    "status": "SUCCESS"
+  }'
+```
+
+## 📖 Full Documentation
+
+See the main [README.md](README.md) for complete documentation, training instructions, and local development setup.
+
+## ⚙️ Configuration
+
+Models are loaded from the `models/` directory. Ensure all required model files are committed to the repository:
+- `job_fail_pipeline_cpu.joblib`
+- `anomaly_autoencoder_cpu.keras`
+- `anomaly_scaler.joblib`
+- `feature_schema.json`
+- `shap_background.npy`
+- `anomaly_features.joblib`
+- `anomaly_threshold.joblib`
+
+## 🔒 Production Ready
+
+- Health check endpoint for monitoring
+- Error handling and validation
+- CORS enabled for cross-origin requests
+- Dockerized for consistent deployment
+- Optimized for Hugging Face Spaces infrastructure

model_utils.py

@@ -0,0 +1,360 @@
+"""
+Model inference utilities with SHAP integration.
+"""
+import numpy as np
+import pandas as pd
+import joblib
+import shap
+from typing import Dict, List, Optional, Tuple
+import os
+from preprocessing import align_schema_df, get_feature_columns
+
+
+class JobFailurePredictor:
+    """Wrapper for job failure prediction model with SHAP explainability."""
+    
+    def __init__(self, model_path: str = None,
+                 background_path: str = None,
+                 schema_path: str = None):
+        """
+        Initialize predictor.
+        
+        Args:
+            model_path: Path to saved pipeline (defaults to env var or 'models/job_fail_pipeline_cpu.joblib')
+            background_path: Path to SHAP background sample (defaults to env var or 'models/shap_background.npy')
+            schema_path: Path to feature schema (defaults to env var or 'models/feature_schema.json')
+        """
+        # Support environment variables for flexible deployment
+        model_base = os.getenv('MODEL_BASE_PATH', 'models')
+        
+        self.model_path = model_path or os.getenv(
+            'JOB_FAIL_MODEL_PATH', 
+            f'{model_base}/job_fail_pipeline_cpu.joblib'
+        )
+        self.background_path = background_path or os.getenv(
+            'SHAP_BACKGROUND_PATH',
+            f'{model_base}/shap_background.npy'
+        )
+        self.schema_path = schema_path or os.getenv(
+            'FEATURE_SCHEMA_PATH',
+            f'{model_base}/feature_schema.json'
+        )
+        
+        if os.path.exists(model_path):
+            self.pipeline = joblib.load(model_path)
+            print(f"Loaded model from {model_path}")
+        else:
+            self.pipeline = None
+            print(f"Warning: Model not found at {model_path}")
+        
+        self.explainer = None
+        self.background = None
+        self._load_shap_background()
+    
+    def _load_shap_background(self):
+        """Load SHAP background sample if available."""
+        if os.path.exists(self.background_path):
+            try:
+                self.background = np.load(self.background_path)
+                # Use generic Explainer for compatibility
+                if self.pipeline is not None:
+                    self.explainer = shap.Explainer(
+                        self.pipeline.predict_proba,
+                        self.background,
+                        feature_names=self._get_feature_names()
+                    )
+                print(f"Loaded SHAP background from {self.background_path}")
+            except Exception as e:
+                print(f"Warning: Could not load SHAP background: {e}")
+                self.explainer = None
+    
+    def _get_feature_names(self) -> List[str]:
+        """Get feature names from pipeline."""
+        if self.pipeline is None:
+            return []
+        
+        try:
+            preprocess = self.pipeline.named_steps['preprocess']
+            num_cols = get_feature_columns()['numeric']
+            cat_cols = get_feature_columns()['categorical']
+            
+            # Get one-hot encoded names
+            cat_encoder = preprocess.named_transformers_['cat']
+            if hasattr(cat_encoder, 'get_feature_names_out'):
+                cat_names = cat_encoder.get_feature_names_out(cat_cols).tolist()
+            else:
+                cat_names = [f"cat_{i}" for i in range(len(cat_cols))]
+            
+            return num_cols + cat_names
+        except Exception:
+            return []
+    
+    def predict(self, data: pd.DataFrame, explain: bool = False) -> Dict:
+        """
+        Predict job failure probability.
+        
+        Args:
+            data: DataFrame with job features
+            explain: Whether to compute SHAP explanations
+        
+        Returns:
+            Dictionary with predictions and optional explanations
+        """
+        if self.pipeline is None:
+            return {
+                'fail_probability': 0.5,
+                'risk_level': 'UNKNOWN',
+                'error': 'Model not loaded'
+            }
+        
+        try:
+            # Align to schema
+            data = align_schema_df(data, self.schema_path)
+            
+            # Get feature columns
+            feature_cols = get_feature_columns()
+            num_cols = feature_cols['numeric']
+            cat_cols = feature_cols['categorical']
+            
+            X = data[num_cols + cat_cols].copy()
+            
+            # Predict
+            proba = self.pipeline.predict_proba(X)[:, 1]
+            fail_prob = float(proba[0]) if len(proba) == 1 else float(proba.mean())
+            
+            # Determine risk level
+            if fail_prob >= 0.8:
+                risk_level = 'CRITICAL'
+            elif fail_prob >= 0.5:
+                risk_level = 'MEDIUM'
+            elif fail_prob >= 0.3:
+                risk_level = 'LOW'
+            else:
+                risk_level = 'MINIMAL'
+            
+            result = {
+                'fail_probability': fail_prob,
+                'risk_level': risk_level
+            }
+            
+            # Add SHAP explanation if requested
+            if explain and self.explainer is not None:
+                try:
+                    # Preprocess to get encoded features
+                    preprocess = self.pipeline.named_steps['preprocess']
+                    X_encoded = preprocess.transform(X)
+                    
+                    # Compute SHAP values
+                    shap_values = self.explainer(X_encoded)
+                    
+                    # Handle different SHAP output shapes
+                    if hasattr(shap_values, 'values'):
+                        sv = shap_values.values
+                        if len(sv.shape) == 3:  # (n_samples, n_features, n_classes)
+                            sv = sv[:, :, 1]  # Take positive class
+                        elif len(sv.shape) == 2:
+                            sv = sv
+                        else:
+                            sv = sv.flatten()
+                    else:
+                        sv = shap_values
+                    
+                    # Get feature names
+                    feature_names = self._get_feature_names()
+                    if len(sv.shape) == 1:
+                        sv = sv.reshape(1, -1)
+                    
+                    # Get top drivers (absolute values)
+                    if len(sv) > 0:
+                        abs_sv = np.abs(sv[0])
+                        top_indices = np.argsort(abs_sv)[::-1][:5]
+                        
+                        top_drivers = []
+                        for idx in top_indices:
+                            if idx < len(feature_names):
+                                feature_name = feature_names[idx]
+                                shap_val = float(sv[0, idx])
+                                top_drivers.append({
+                                    'feature': feature_name,
+                                    'shap_value': shap_val,
+                                    'effect': 'increase' if shap_val > 0 else 'decrease'
+                                })
+                        
+                        result['top_drivers'] = top_drivers
+                        
+                        # Generate recommended actions
+                        result['recommended_actions'] = self._generate_actions(
+                            top_drivers, fail_prob
+                        )
+                except Exception as e:
+                    print(f"Warning: SHAP explanation failed: {e}")
+                    result['top_drivers'] = []
+            
+            return result
+            
+        except Exception as e:
+            return {
+                'fail_probability': 0.5,
+                'risk_level': 'UNKNOWN',
+                'error': str(e)
+            }
+    
+    def _generate_actions(self, top_drivers: List[Dict], fail_prob: float) -> List[str]:
+        """Generate recommended actions based on top drivers."""
+        actions = []
+        
+        for driver in top_drivers[:3]:
+            feature = driver['feature']
+            effect = driver['effect']
+            
+            if 'failure_rate' in feature:
+                actions.append("Monitor upstream dependencies and recent job history")
+            elif 'duration' in feature:
+                if effect == 'increase':
+                    actions.append("Check for resource constraints or data volume spikes")
+                else:
+                    actions.append("Verify job completed successfully (unusually fast)")
+            elif 'err_msg' in feature:
+                actions.append("Review error logs and investigate root cause")
+            elif 'job_nm' in feature or 'tasksgroup' in feature:
+                actions.append("Check job configuration and dependencies")
+        
+        if fail_prob >= 0.8:
+            actions.append("Consider immediate intervention or rerun")
+        elif fail_prob >= 0.5:
+            actions.append("Increase monitoring frequency")
+        
+        # Deduplicate
+        return list(dict.fromkeys(actions))[:5]
+
+
+class AnomalyDetector:
+    """Wrapper for anomaly detection model."""
+    
+    def __init__(self, model_path: str = None,
+                 scaler_path: str = None,
+                 feature_path: str = None,
+                 threshold_path: str = None):
+        """
+        Initialize anomaly detector.
+        
+        Args:
+            model_path: Path to saved autoencoder (defaults to env var or 'models/anomaly_autoencoder_cpu.keras')
+            scaler_path: Path to saved scaler (defaults to env var or 'models/anomaly_scaler.joblib')
+            feature_path: Path to saved feature list (defaults to env var or 'models/anomaly_features.joblib')
+            threshold_path: Path to saved threshold (defaults to env var or 'models/anomaly_threshold.joblib')
+        """
+        # Support environment variables for flexible deployment
+        model_base = os.getenv('MODEL_BASE_PATH', 'models')
+        
+        self.model_path = model_path or os.getenv(
+            'ANOMALY_MODEL_PATH',
+            f'{model_base}/anomaly_autoencoder_cpu.keras'
+        )
+        self.scaler_path = scaler_path or os.getenv(
+            'ANOMALY_SCALER_PATH',
+            f'{model_base}/anomaly_scaler.joblib'
+        )
+        self.feature_path = feature_path or os.getenv(
+            'ANOMALY_FEATURE_PATH',
+            f'{model_base}/anomaly_features.joblib'
+        )
+        self.threshold_path = threshold_path or os.getenv(
+            'ANOMALY_THRESHOLD_PATH',
+            f'{model_base}/anomaly_threshold.joblib'
+        )
+        
+        if os.path.exists(model_path):
+            from tensorflow import keras
+            self.model = keras.models.load_model(model_path)
+            print(f"Loaded autoencoder from {model_path}")
+        else:
+            self.model = None
+            print(f"Warning: Model not found at {model_path}")
+        
+        if os.path.exists(scaler_path):
+            self.scaler = joblib.load(scaler_path)
+            print(f"Loaded scaler from {scaler_path}")
+        else:
+            self.scaler = None
+        
+        if os.path.exists(feature_path):
+            self.feature_list = joblib.load(feature_path)
+            print(f"Loaded feature list from {feature_path}")
+        else:
+            self.feature_list = get_feature_columns()['anomaly_numeric']
+        
+        if os.path.exists(threshold_path):
+            self.global_threshold = joblib.load(threshold_path)
+            print(f"Loaded threshold from {threshold_path}")
+        else:
+            self.global_threshold = 0.01
+    
+    def detect(self, features: Dict, threshold: Optional[float] = None) -> Dict:
+        """
+        Detect anomaly in feature vector.
+        
+        Args:
+            features: Dictionary of feature values
+            threshold: Optional custom threshold (overrides default)
+        
+        Returns:
+            Dictionary with anomaly detection results
+        """
+        if self.model is None or self.scaler is None:
+            return {
+                'reconstruction_error': 0.0,
+                'is_anomaly': False,
+                'error': 'Model not loaded'
+            }
+        
+        try:
+            # Build feature vector
+            feature_vec = []
+            for feat in self.feature_list:
+                value = features.get(feat, 0.0)
+                try:
+                    feature_vec.append(float(value))
+                except (ValueError, TypeError):
+                    feature_vec.append(0.0)
+            
+            feature_vec = np.array(feature_vec).reshape(1, -1)
+            
+            # Scale
+            feature_scaled = self.scaler.transform(feature_vec)
+            
+            # Reconstruct
+            reconstructed = self.model.predict(feature_scaled, verbose=0)
+            
+            # Compute reconstruction error
+            recon_error = np.mean((feature_scaled - reconstructed) ** 2)
+            
+            # Determine if anomaly
+            thresh = threshold if threshold is not None else self.global_threshold
+            is_anomaly = recon_error > thresh
+            
+            # Compute per-feature errors for top drivers
+            per_feature_errors = np.square(feature_scaled - reconstructed).flatten()
+            top_indices = np.argsort(per_feature_errors)[::-1][:3]
+            
+            top_drivers = []
+            for idx in top_indices:
+                if idx < len(self.feature_list):
+                    top_drivers.append({
+                        'feature': self.feature_list[idx],
+                        'error': float(per_feature_errors[idx])
+                    })
+            
+            return {
+                'reconstruction_error': float(recon_error),
+                'is_anomaly': bool(is_anomaly),
+                'threshold': float(thresh),
+                'top_drivers': top_drivers
+            }
+            
+        except Exception as e:
+            return {
+                'reconstruction_error': 0.0,
+                'is_anomaly': False,
+                'error': str(e)
+            }

preprocessing.py

@@ -0,0 +1,175 @@
+"""
+Data preprocessing and feature engineering utilities for job failure prediction.
+"""
+import pandas as pd
+import numpy as np
+from typing import Dict, List, Optional
+import json
+
+
+def parse_duration(x) -> float:
+    """Parse duration string (HH:MM:SS) or numeric to seconds."""
+    if pd.isna(x):
+        return 0.0
+    try:
+        if isinstance(x, str):
+            parts = x.split(':')
+            if len(parts) == 3:
+                h, m, s = map(int, parts)
+                return h * 3600 + m * 60 + s
+            return float(x)
+        return float(x)
+    except (ValueError, AttributeError):
+        return 0.0
+
+
+def engineer_features(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Engineer features from raw job data.
+    
+    Args:
+        df: DataFrame with columns: zone, job_nm, tasksgroup_nm, round_time,
+            job_start_time, job_end_time, duration, status, err_msg, zeppelin,
+            ictrl_dt, start_ictrl_dt, end_ictrl_dt
+    
+    Returns:
+        DataFrame with engineered features
+    """
+    df = df.copy()
+    
+    # Parse timestamps
+    df['job_start_time'] = pd.to_datetime(df['job_start_time'], errors='coerce')
+    df['job_end_time'] = pd.to_datetime(df['job_end_time'], errors='coerce')
+    
+    # Parse duration to seconds
+    if 'duration' in df.columns:
+        df['duration_sec'] = df['duration'].apply(parse_duration)
+    else:
+        df['duration_sec'] = 0.0
+    
+    df['duration_sec'] = df['duration_sec'].fillna(0.0)
+    
+    # Ground truth label
+    # Handle various success statuses: 'SUCCESS', 'SUCCEED', 'SUCCEEDED'
+    # Everything else (FAILED, ABORT-AUTO, RUNNING, etc.) is considered a failure
+    status_upper = df['status'].fillna('').str.upper()
+    success_statuses = ['SUCCESS', 'SUCCEED', 'SUCCEEDED']
+    df['is_failed'] = (~status_upper.isin(success_statuses)).astype(int)
+    
+    # Time features
+    df['run_hour'] = df['job_start_time'].dt.hour.fillna(0).astype(int)
+    df['run_dow'] = df['job_start_time'].dt.dayofweek.fillna(0).astype(int)  # 0=Mon, 6=Sun
+    df['is_weekend'] = (df['run_dow'] >= 5).astype(int)
+    
+    # Cyclical encoding for hour
+    df['hour_sin'] = np.sin(2 * np.pi * df['run_hour'] / 24)
+    df['hour_cos'] = np.cos(2 * np.pi * df['run_hour'] / 24)
+    
+    # Error message features
+    df['err_msg_len'] = df['err_msg'].fillna('').str.len()
+    df['has_err_msg'] = (df['err_msg_len'] > 0).astype(int)
+    
+    # Zeppelin flag
+    df['is_zeppelin'] = df['zeppelin'].notna().astype(int)
+    
+    # Job-level rolling statistics (per job_nm)
+    df = df.sort_values(['job_nm', 'job_start_time']).reset_index(drop=True)
+    
+    df['failure_rate_7'] = df.groupby('job_nm')['is_failed'].transform(
+        lambda s: s.rolling(7, min_periods=1).mean()
+    )
+    
+    df['avg_duration_7'] = df.groupby('job_nm')['duration_sec'].transform(
+        lambda s: s.rolling(7, min_periods=1).mean()
+    )
+    
+    # Duration z-score (relative to rolling average)
+    df['duration_zscore'] = (
+        (df['duration_sec'] - df['avg_duration_7']) / 
+        df['avg_duration_7'].replace(0, 1)
+    )
+    df['duration_zscore'] = df['duration_zscore'].fillna(0.0)
+    
+    return df
+
+
+def get_feature_columns() -> Dict[str, List[str]]:
+    """Return feature column definitions."""
+    return {
+        'numeric': [
+            'duration_sec',
+            'duration_zscore',
+            'avg_duration_7',
+            'failure_rate_7',
+            'err_msg_len',
+            'hour_sin',
+            'hour_cos'
+        ],
+        'categorical': [
+            'job_nm',
+            'tasksgroup_nm',
+            'zone',
+            'is_zeppelin',
+            'is_weekend'
+        ],
+        'anomaly_numeric': [
+            'duration_sec',
+            'duration_zscore',
+            'avg_duration_7',
+            'failure_rate_7',
+            'err_msg_len',
+            'hour_sin',
+            'hour_cos'
+        ]
+    }
+
+
+def save_feature_schema(output_path: str = 'models/feature_schema.json'):
+    """Save feature schema to JSON file."""
+    import os
+    os.makedirs(os.path.dirname(output_path), exist_ok=True)
+    
+    schema = {
+        'feature_columns': get_feature_columns(),
+        'required_fields': [
+            'zone', 'job_nm', 'tasksgroup_nm', 'job_start_time', 
+            'duration', 'status', 'err_msg', 'zeppelin'
+        ]
+    }
+    
+    with open(output_path, 'w') as f:
+        json.dump(schema, f, indent=2)
+    
+    return schema
+
+
+def align_schema_df(df: pd.DataFrame, schema_path: str = 'models/feature_schema.json') -> pd.DataFrame:
+    """
+    Align DataFrame to feature schema, filling missing columns with defaults.
+    
+    Args:
+        df: Input DataFrame
+        schema_path: Path to feature schema JSON
+    
+    Returns:
+        Aligned DataFrame
+    """
+    try:
+        with open(schema_path, 'r') as f:
+            schema = json.load(f)
+    except FileNotFoundError:
+        # If schema doesn't exist, engineer features and create it
+        df = engineer_features(df)
+        schema = save_feature_schema(schema_path)
+    
+    # Ensure all required numeric and categorical columns exist
+    all_features = schema['feature_columns']['numeric'] + schema['feature_columns']['categorical']
+    
+    for col in all_features:
+        if col not in df.columns:
+            if col in schema['feature_columns']['numeric']:
+                df[col] = 0.0
+            else:
+                df[col] = ''
+    
+    return df