SETUP.md

Setup Instructions

1. Install Dependencies

Option A: Using pip (recommended)

pip install -r requirements.txt

Option B: Using virtual environment (recommended for isolation)

# Create virtual environment
python -m venv venv

# Activate virtual environment
# On Windows (PowerShell):
.\venv\Scripts\Activate.ps1
# On Windows (CMD):
venv\Scripts\activate.bat
# On Linux/Mac:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

2. Prepare Data

Place your CSV files in the data/ directory:

mkdir -p data
# Copy your CSV files to data/

3. Train Models

Train Job Failure Prediction Model

# Option 1: Use all CSV files in data/ directory (recommended)
python train_job_failure.py

# Option 2: Specify files explicitly
python train_job_failure.py data/true_export_report_20260120.csv data/true_export_report_20260121.csv

# Option 3: Use glob pattern (works in PowerShell and bash)
python train_job_failure.py data/*.csv

Train Anomaly Detection Model

# Option 1: Use all CSV files in data/ directory (recommended)
python train_anomaly.py

# Option 2: Specify files explicitly
python train_anomaly.py data/true_export_report_20260120.csv data/true_export_report_20260121.csv

# Option 3: Use glob pattern
python train_anomaly.py data/*.csv

4. Verify Models

After training, check that models were created:

ls models/

You should see:

• job_fail_pipeline_cpu.joblib
• anomaly_autoencoder_cpu.keras
• anomaly_scaler.joblib
• feature_schema.json
• shap_background.npy
• anomaly_features.joblib
• anomaly_threshold.joblib

5. Run the Service

Local Development

uvicorn app:app --host 0.0.0.0 --port 8000

Docker

docker-compose up --build

Troubleshooting

"ModuleNotFoundError: No module named 'tensorflow'"

Install dependencies:

pip install -r requirements.txt

"No data files found!"

1. Check that CSV files exist in data/ directory:

   ls data/
   

2. Use explicit file paths:

   python train_job_failure.py data/true_export_report_20260120.csv
   

PowerShell glob pattern not working

The scripts now handle glob patterns internally. You can use:

python train_job_failure.py data/*.csv

Or just run without arguments to use all CSV files in data/:

python train_job_failure.py