# Fraud Detection on PaySim — End-to-End ML Pipeline

A production-style machine learning pipeline that detects fraudulent mobile-money
transactions in the [PaySim synthetic dataset](https://www.kaggle.com/datasets/ealaxi/paysim1)
(~6.3M transactions). The project covers the full lifecycle: data validation,
feature engineering, model training with experiment tracking, and model serving
through both a REST API and a web UI.

> **Focus:** this repo is less about squeezing out the last decimal of accuracy and
> more about doing the *engineering* around a model correctly — no data leakage,
> reproducible training, tracked experiments, saved artifacts, and a real serving layer.

---

## Results

Evaluated on a **chronologically held-out test set** (later time steps the model never
saw during training), so the numbers reflect genuine generalization rather than
memorization.

| Metric | @ threshold 0.50 | @ tuned threshold (0.64) |
|---|---|---|
| **PR-AUC** | **0.90** | 0.90 |
| Precision | 0.83 | 0.95 |
| Recall | 0.84 | 0.79 |
| F1 | 0.83 | 0.87 |

*PR-AUC (area under the precision–recall curve) is the headline metric — accuracy is
misleading here because only ~0.2–4% of transactions are fraudulent.*

**Known failure mode:** the model catches `TRANSFER` fraud almost perfectly (>99%
confidence) but is weaker on `CASH_OUT` fraud. In PaySim, fraud is typically a
`TRANSFER` of stolen funds *followed by* a `CASH_OUT` of the same money — the `TRANSFER`
leg carries the obvious "account drained to zero" signature, while the `CASH_OUT` leg
looks much like a legitimate withdrawal. Improving `CASH_OUT` recall (e.g. features
that link a cash-out back to a preceding transfer) is the clearest next modelling step.

> ⚠️ **Honesty note:** the tuned threshold (0.64) was selected on the test set, so its
> precision is mildly optimistic. The @0.50 numbers are the trustworthy ones. The test
> window is also more fraud-dense than the training window, so real-world precision at a
> lower fraud base rate would be lower.

---

## Approach

**Avoiding data leakage** — the most common mistake on this dataset:
- **Chronological split** by transaction `step` (train on earlier steps, test on later),
  mimicking how a fraud model is used in production: trained on the past, scoring the future.
- The feature scaler is **fit on the training split only** and reused at inference.
- The class-imbalance weight (`pos_weight`) is computed from the **training split only**.

**Feature engineering** (in `src/data/preprocess.py`):
- `errorBalanceOrig` / `errorBalanceDest` — accounting inconsistencies between stated
  balances and transaction amounts (the highest-signal engineered features).
- `orig_balance_missing` / `dest_balance_missing` — flags for placeholder-zero balances.
- One-hot encoded transaction type; only `TRANSFER` and `CASH_OUT` are kept, as these
  are the only types where fraud occurs in PaySim.

**Model:** a small PyTorch MLP (`FraudMLP`) with BatchNorm and Dropout, trained with
`BCEWithLogitsLoss` weighted for the severe class imbalance.

**Threshold tuning:** instead of a naive 0.5 cutoff, the decision threshold is selected
from the precision–recall curve to balance precision and recall (F1-optimal).

---

## Tech stack

| Area | Tools |
|---|---|
| Data / features | pandas, scikit-learn |
| Data validation | Great Expectations |
| Modelling | PyTorch |
| Experiment tracking | MLflow |
| Serving | FastAPI, Gradio, Uvicorn, Pydantic |

---

## Project structure

```
fraud-paysim/
├── src/
│   ├── data/
│   │   ├── load_data.py       # load the raw CSV
│   │   └── preprocess.py      # cleaning + feature engineering
│   ├── features/
│   │   └── build_features.py  # chronological split, scaling, DataLoaders
│   ├── utils/
│   │   └── validate_data.py   # Great Expectations checks (gate before training)
│   ├── models/
│   │   ├── model.py           # FraudMLP architecture
│   │   ├── train.py           # training loop + MLflow logging + test metrics
│   │   └── evaluate.py        # evaluation + threshold tuning helpers
│   └── app/
│       ├── inference.py       # shared predict(): raw transaction -> probability
│       └── main.py            # FastAPI API (/predict) + Gradio UI (/ui)
├── scripts/
│   └── run_pipeline.py        # end-to-end: load -> validate -> train -> save
├── requirements.txt
└── Readme.md
```

---

## Setup

**1. Get the data.** The dataset is not included in the repo (it's ~470 MB). Download
"Synthetic Financial Datasets For Fraud Detection" from
[Kaggle](https://www.kaggle.com/datasets/ealaxi/paysim1) and place the CSV at:

```
fraud-paysim/data/Synthetic_Financial_datasets_log.csv
```

**2. Create a virtual environment and install dependencies:**

```bash
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

pip install -r requirements.txt
```

---

## Usage

### Train the model

Runs the full pipeline (load → validate → feature-build → train → evaluate → save artifacts):

```bash
python scripts/run_pipeline.py
```

This logs metrics to MLflow and writes three artifacts to `artifacts/`:
`model.pt` (weights), `scaler.joblib` (fitted scaler), and `metadata.json`
(input dimension + exact feature column order). These three together are what the
serving layer needs to reproduce training-time preprocessing exactly.

Inspect experiments with:

```bash
mlflow ui
```

### Serve the model

```bash
uvicorn src.app.main:app --reload
```

Then open:
- **http://localhost:8000/docs** — interactive REST API (Swagger UI) for the `/predict` endpoint
- **http://localhost:8000/ui** — Gradio web form for manual predictions

**Example request** (a known fraud pattern — full balance transferred out, origin drained to zero):

```json
{
  "step": 1,
  "type": "TRANSFER",
  "amount": 181.0,
  "oldbalanceOrg": 181.0,
  "newbalanceOrig": 0.0,
  "oldbalanceDest": 0.0,
  "newbalanceDest": 0.0
}
```

Response:

```json
{ "fraud_probability": 0.9952, "is_fraud": 1 }
```

---

## Design notes

- **Shared inference path.** Both the API and the Gradio UI call the same
  `predict()` function, so there is a single source of truth for how a raw transaction
  is turned into a prediction — no risk of training/serving skew between the two faces.
- **Raw in, prediction out.** The serving layer accepts *raw* transaction fields and
  reconstructs the engineered features internally using the same `preprocess_data`
  used in training. Clients never compute features themselves.

---