Spaces:
Sleeping
Sleeping
| # Fraud Detection on PaySim — End-to-End ML Pipeline | |
| A production-style machine learning pipeline that detects fraudulent mobile-money | |
| transactions in the [PaySim synthetic dataset](https://www.kaggle.com/datasets/ealaxi/paysim1) | |
| (~6.3M transactions). The project covers the full lifecycle: data validation, | |
| feature engineering, model training with experiment tracking, and model serving | |
| through both a REST API and a web UI. | |
| > **Focus:** this repo is less about squeezing out the last decimal of accuracy and | |
| > more about doing the *engineering* around a model correctly — no data leakage, | |
| > reproducible training, tracked experiments, saved artifacts, and a real serving layer. | |
| --- | |
| ## Results | |
| Evaluated on a **chronologically held-out test set** (later time steps the model never | |
| saw during training), so the numbers reflect genuine generalization rather than | |
| memorization. | |
| | Metric | @ threshold 0.50 | @ tuned threshold (0.64) | | |
| |---|---|---| | |
| | **PR-AUC** | **0.90** | 0.90 | | |
| | Precision | 0.83 | 0.95 | | |
| | Recall | 0.84 | 0.79 | | |
| | F1 | 0.83 | 0.87 | | |
| *PR-AUC (area under the precision–recall curve) is the headline metric — accuracy is | |
| misleading here because only ~0.2–4% of transactions are fraudulent.* | |
| **Known failure mode:** the model catches `TRANSFER` fraud almost perfectly (>99% | |
| confidence) but is weaker on `CASH_OUT` fraud. In PaySim, fraud is typically a | |
| `TRANSFER` of stolen funds *followed by* a `CASH_OUT` of the same money — the `TRANSFER` | |
| leg carries the obvious "account drained to zero" signature, while the `CASH_OUT` leg | |
| looks much like a legitimate withdrawal. Improving `CASH_OUT` recall (e.g. features | |
| that link a cash-out back to a preceding transfer) is the clearest next modelling step. | |
| > ⚠️ **Honesty note:** the tuned threshold (0.64) was selected on the test set, so its | |
| > precision is mildly optimistic. The @0.50 numbers are the trustworthy ones. The test | |
| > window is also more fraud-dense than the training window, so real-world precision at a | |
| > lower fraud base rate would be lower. | |
| --- | |
| ## Approach | |
| **Avoiding data leakage** — the most common mistake on this dataset: | |
| - **Chronological split** by transaction `step` (train on earlier steps, test on later), | |
| mimicking how a fraud model is used in production: trained on the past, scoring the future. | |
| - The feature scaler is **fit on the training split only** and reused at inference. | |
| - The class-imbalance weight (`pos_weight`) is computed from the **training split only**. | |
| **Feature engineering** (in `src/data/preprocess.py`): | |
| - `errorBalanceOrig` / `errorBalanceDest` — accounting inconsistencies between stated | |
| balances and transaction amounts (the highest-signal engineered features). | |
| - `orig_balance_missing` / `dest_balance_missing` — flags for placeholder-zero balances. | |
| - One-hot encoded transaction type; only `TRANSFER` and `CASH_OUT` are kept, as these | |
| are the only types where fraud occurs in PaySim. | |
| **Model:** a small PyTorch MLP (`FraudMLP`) with BatchNorm and Dropout, trained with | |
| `BCEWithLogitsLoss` weighted for the severe class imbalance. | |
| **Threshold tuning:** instead of a naive 0.5 cutoff, the decision threshold is selected | |
| from the precision–recall curve to balance precision and recall (F1-optimal). | |
| --- | |
| ## Tech stack | |
| | Area | Tools | | |
| |---|---| | |
| | Data / features | pandas, scikit-learn | | |
| | Data validation | Great Expectations | | |
| | Modelling | PyTorch | | |
| | Experiment tracking | MLflow | | |
| | Serving | FastAPI, Gradio, Uvicorn, Pydantic | | |
| --- | |
| ## Project structure | |
| ``` | |
| fraud-paysim/ | |
| ├── src/ | |
| │ ├── data/ | |
| │ │ ├── load_data.py # load the raw CSV | |
| │ │ └── preprocess.py # cleaning + feature engineering | |
| │ ├── features/ | |
| │ │ └── build_features.py # chronological split, scaling, DataLoaders | |
| │ ├── utils/ | |
| │ │ └── validate_data.py # Great Expectations checks (gate before training) | |
| │ ├── models/ | |
| │ │ ├── model.py # FraudMLP architecture | |
| │ │ ├── train.py # training loop + MLflow logging + test metrics | |
| │ │ └── evaluate.py # evaluation + threshold tuning helpers | |
| │ └── app/ | |
| │ ├── inference.py # shared predict(): raw transaction -> probability | |
| │ └── main.py # FastAPI API (/predict) + Gradio UI (/ui) | |
| ├── scripts/ | |
| │ └── run_pipeline.py # end-to-end: load -> validate -> train -> save | |
| ├── requirements.txt | |
| └── Readme.md | |
| ``` | |
| --- | |
| ## Setup | |
| **1. Get the data.** The dataset is not included in the repo (it's ~470 MB). Download | |
| "Synthetic Financial Datasets For Fraud Detection" from | |
| [Kaggle](https://www.kaggle.com/datasets/ealaxi/paysim1) and place the CSV at: | |
| ``` | |
| fraud-paysim/data/Synthetic_Financial_datasets_log.csv | |
| ``` | |
| **2. Create a virtual environment and install dependencies:** | |
| ```bash | |
| python -m venv .venv | |
| # Windows | |
| .venv\Scripts\activate | |
| # macOS / Linux | |
| source .venv/bin/activate | |
| pip install -r requirements.txt | |
| ``` | |
| --- | |
| ## Usage | |
| ### Train the model | |
| Runs the full pipeline (load → validate → feature-build → train → evaluate → save artifacts): | |
| ```bash | |
| python scripts/run_pipeline.py | |
| ``` | |
| This logs metrics to MLflow and writes three artifacts to `artifacts/`: | |
| `model.pt` (weights), `scaler.joblib` (fitted scaler), and `metadata.json` | |
| (input dimension + exact feature column order). These three together are what the | |
| serving layer needs to reproduce training-time preprocessing exactly. | |
| Inspect experiments with: | |
| ```bash | |
| mlflow ui | |
| ``` | |
| ### Serve the model | |
| ```bash | |
| uvicorn src.app.main:app --reload | |
| ``` | |
| Then open: | |
| - **http://localhost:8000/docs** — interactive REST API (Swagger UI) for the `/predict` endpoint | |
| - **http://localhost:8000/ui** — Gradio web form for manual predictions | |
| **Example request** (a known fraud pattern — full balance transferred out, origin drained to zero): | |
| ```json | |
| { | |
| "step": 1, | |
| "type": "TRANSFER", | |
| "amount": 181.0, | |
| "oldbalanceOrg": 181.0, | |
| "newbalanceOrig": 0.0, | |
| "oldbalanceDest": 0.0, | |
| "newbalanceDest": 0.0 | |
| } | |
| ``` | |
| Response: | |
| ```json | |
| { "fraud_probability": 0.9952, "is_fraud": 1 } | |
| ``` | |
| --- | |
| ## Design notes | |
| - **Shared inference path.** Both the API and the Gradio UI call the same | |
| `predict()` function, so there is a single source of truth for how a raw transaction | |
| is turned into a prediction — no risk of training/serving skew between the two faces. | |
| - **Raw in, prediction out.** The serving layer accepts *raw* transaction fields and | |
| reconstructs the engineered features internally using the same `preprocess_data` | |
| used in training. Clients never compute features themselves. | |
| --- | |