Readme.md

Fraud Detection on PaySim — End-to-End ML Pipeline

A production-style machine learning pipeline that detects fraudulent mobile-money transactions in the PaySim synthetic dataset (~6.3M transactions). The project covers the full lifecycle: data validation, feature engineering, model training with experiment tracking, and model serving through both a REST API and a web UI.

Focus: this repo is less about squeezing out the last decimal of accuracy and more about doing the engineering around a model correctly — no data leakage, reproducible training, tracked experiments, saved artifacts, and a real serving layer.

---

Results

Evaluated on a chronologically held-out test set (later time steps the model never saw during training), so the numbers reflect genuine generalization rather than memorization.

Metric	@ threshold 0.50	@ tuned threshold (0.64)
PR-AUC	0.90	0.90
Precision	0.83	0.95
Recall	0.84	0.79
F1	0.83	0.87

PR-AUC (area under the precision–recall curve) is the headline metric — accuracy is misleading here because only ~0.2–4% of transactions are fraudulent.

Known failure mode: the model catches TRANSFER fraud almost perfectly (>99% confidence) but is weaker on CASH_OUT fraud. In PaySim, fraud is typically a TRANSFER of stolen funds followed by a CASH_OUT of the same money — the TRANSFER leg carries the obvious "account drained to zero" signature, while the CASH_OUT leg looks much like a legitimate withdrawal. Improving CASH_OUT recall (e.g. features that link a cash-out back to a preceding transfer) is the clearest next modelling step.

⚠️ Honesty note: the tuned threshold (0.64) was selected on the test set, so its precision is mildly optimistic. The @0.50 numbers are the trustworthy ones. The test window is also more fraud-dense than the training window, so real-world precision at a lower fraud base rate would be lower.

---

Approach

Avoiding data leakage — the most common mistake on this dataset:

• Chronological split by transaction step (train on earlier steps, test on later), mimicking how a fraud model is used in production: trained on the past, scoring the future.
• The feature scaler is fit on the training split only and reused at inference.
• The class-imbalance weight (pos_weight) is computed from the training split only.

Feature engineering (in src/data/preprocess.py):

• errorBalanceOrig / errorBalanceDest — accounting inconsistencies between stated balances and transaction amounts (the highest-signal engineered features).
• orig_balance_missing / dest_balance_missing — flags for placeholder-zero balances.
• One-hot encoded transaction type; only TRANSFER and CASH_OUT are kept, as these are the only types where fraud occurs in PaySim.

Model: a small PyTorch MLP (FraudMLP) with BatchNorm and Dropout, trained with BCEWithLogitsLoss weighted for the severe class imbalance.

Threshold tuning: instead of a naive 0.5 cutoff, the decision threshold is selected from the precision–recall curve to balance precision and recall (F1-optimal).

---

Tech stack

Area	Tools
Data / features	pandas, scikit-learn
Data validation	Great Expectations
Modelling	PyTorch
Experiment tracking	MLflow
Serving	FastAPI, Gradio, Uvicorn, Pydantic

---

Project structure

fraud-paysim/
├── src/
│   ├── data/
│   │   ├── load_data.py       # load the raw CSV
│   │   └── preprocess.py      # cleaning + feature engineering
│   ├── features/
│   │   └── build_features.py  # chronological split, scaling, DataLoaders
│   ├── utils/
│   │   └── validate_data.py   # Great Expectations checks (gate before training)
│   ├── models/
│   │   ├── model.py           # FraudMLP architecture
│   │   ├── train.py           # training loop + MLflow logging + test metrics
│   │   └── evaluate.py        # evaluation + threshold tuning helpers
│   └── app/
│       ├── inference.py       # shared predict(): raw transaction -> probability
│       └── main.py            # FastAPI API (/predict) + Gradio UI (/ui)
├── scripts/
│   └── run_pipeline.py        # end-to-end: load -> validate -> train -> save
├── requirements.txt
└── Readme.md

---

Setup

1. Get the data. The dataset is not included in the repo (it's ~470 MB). Download "Synthetic Financial Datasets For Fraud Detection" from Kaggle and place the CSV at:

fraud-paysim/data/Synthetic_Financial_datasets_log.csv

2. Create a virtual environment and install dependencies:

python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt

---

Usage

Train the model

Runs the full pipeline (load → validate → feature-build → train → evaluate → save artifacts):

python scripts/run_pipeline.py

This logs metrics to MLflow and writes three artifacts to artifacts/: model.pt (weights), scaler.joblib (fitted scaler), and metadata.json (input dimension + exact feature column order). These three together are what the serving layer needs to reproduce training-time preprocessing exactly.

Inspect experiments with:

mlflow ui

Serve the model

uvicorn src.app.main:app --reload

Then open:

• http://localhost:8000/docs — interactive REST API (Swagger UI) for the /predict endpoint
• http://localhost:8000/ui — Gradio web form for manual predictions

Example request (a known fraud pattern — full balance transferred out, origin drained to zero):

{
  "step": 1,
  "type": "TRANSFER",
  "amount": 181.0,
  "oldbalanceOrg": 181.0,
  "newbalanceOrig": 0.0,
  "oldbalanceDest": 0.0,
  "newbalanceDest": 0.0
}

Response:

{ "fraud_probability": 0.9952, "is_fraud": 1 }

---

Design notes

• Shared inference path. Both the API and the Gradio UI call the same predict() function, so there is a single source of truth for how a raw transaction is turned into a prediction — no risk of training/serving skew between the two faces.
• Raw in, prediction out. The serving layer accepts raw transaction fields and reconstructs the engineered features internally using the same preprocess_data used in training. Clients never compute features themselves.

---