Spaces:
Runtime error
title: MuleGuard
emoji: 🛡️
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: true
license: mit
short_description: Real-time AI/ML detection of suspicious mule accounts
MuleGuard — AI/ML Detection of Suspicious Mule Accounts
BOI Hackathon · Problem Statement 2 — AI/ML-Based Classification of Suspicious Mule Accounts
An end-to-end system that ingests account/transaction features, scores every account for mule-risk in real time, explains every alert, and streams alerts to a fraud-analyst console — directly targeting the circulation of fraudulent proceeds through mule accounts.
1. Problem
Banks face growing cyber-enabled fraud where mule accounts receive, transfer, and conceal fraudulent funds across channels. Rule-based monitoring misses evolving patterns. We must learn behavioral & transactional patterns from the provided data to classify suspicious/mule accounts, with anomaly detection, predictive risk scoring, and intelligent alerts.
Dataset: data/DataSet.csv — 9,082 accounts × 3,924 anonymized features (F1–F3924). Target = F3924. Severe imbalance: 9,001 legit (0) vs 81 mule (1) → 0.89% positive. 18 features flagged by the bank as high-signal.
2. Our Solution — "MuleGuard"
A supervised gradient-boosted classifier fused with an unsupervised anomaly score, wrapped in a real-time scoring service, a live feed simulator, and an analyst dashboard:
DataSet.csv ─▶ Feature Pipeline ─▶ Model + Calibrator + SHAP ─┐
├─▶ artifacts/
Feed Simulator ─(streams accounts/tickets)─▶ FastAPI /score ───┘
│ risk 0-100, tier, reason codes, alert
▼
Streamlit Analyst Console
Why this wins: proven imbalanced-tabular method + novelty detection (catches evolving fraud) + full explainability (auditable alerts) + a working real-time demo that maps to the regulatory-feed vision.
3. Architecture & Engineering Approach
- Anti-leakage discipline — all transforms fit inside CV folds / train only.
- Honest metrics — PR-AUC, ROC-AUC, recall@precision, F2 (never accuracy).
- Repeated stratified CV — robust estimates despite only 81 positives.
- Threshold tuning + probability calibration — meaningful 0–100 risk scores.
- SHAP reason codes — every alert is explainable.
Built using a custom .claude/ toolkit (agents + skills) authored before implementation — see §6.
4. Repository Layout
.claude/ custom agents + skills (incl. vendored superpowers & get-shit-done)
data/ DataSet.csv (gitignored)
src/data/ load, profile, clean
src/features/ selection + engineering pipeline
src/models/ train, evaluate, calibrate, explain
src/api/ FastAPI scoring service
src/simulator/ real-time feed simulator
dashboard/ Streamlit analyst console
artifacts/ trained model, calibrator, threshold, feature list, SHAP explainer
reports/ data profile, feature selection, model card, figures
docs/ architecture diagrams + spec
deck/ pitch deck (.pptx)
5. How to Run
python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python -m src.data.profile # 1. profile data -> reports/data_profile.md
.venv/bin/python -m src.features.build # 2. build features -> artifacts/
.venv/bin/python -m src.models.train # 3. train + evaluate -> artifacts/, reports/model_report.md
.venv/bin/python -m src.models.explain # 4. SHAP explainability -> artifacts/, reports/figures/
.venv/bin/uvicorn src.api.main:app --port 8000 # 5. scoring API
.venv/bin/python -m src.simulator.feed # 6. stream live alerts
.venv/bin/streamlit run dashboard/app.py # 7. analyst console
6. Custom .claude/ Toolkit (built first, then used)
Agents (.claude/agents/): data-profiler, feature-engineer, ml-trainer, explainability-analyst, fraud-domain-expert, demo-builder.
Skills (.claude/skills/): imbalanced-fraud-classification, fraud-feature-engineering (authored), plus superpowers and get-shit-done vendored from GitHub as-is.
7. Progress Tracker
| # | Phase | Status |
|---|---|---|
| 0 | Scaffold: structure, venv, deps, .claude/ agents & skills |
✅ Done |
| 1 | Data profiling (reports/data_profile.md) |
✅ Done |
| 2 | Feature selection + engineering pipeline (103 features) | ✅ Done |
| 3 | Model training, CV, calibration, threshold | ✅ Done |
| 4 | SHAP explainability | ✅ Done |
| 5 | FastAPI scoring service | ✅ Done |
| 6 | Real-time feed simulator | ✅ Done |
| 7 | Streamlit analyst dashboard | ✅ Done |
| 8 | Architecture diagrams (docs/) |
✅ Done |
| 9 | Pitch deck (deck/MuleGuard_Pitch.pptx) |
✅ Done |
8. Results
Model: Calibrated LightGBM fused with an Isolation Forest anomaly score · 103 features selected from 3,924.
Leakage defense (the rigor that makes the numbers trustworthy)
Two label-leaking features were detected and removed — without this, the model "achieved" a meaningless 100%:
F3912— a binary fraud-flag aligned ~perfectly with the label (single-feature AUC 0.987).F2230— observation month; all legit accounts in Oct-25, all mules in Sep/Nov/Dec-25 (temporal artifact, AUC 1.000).
Headline metrics
| Metric | Value |
|---|---|
| Cross-validated PR-AUC (repeated stratified) | 0.882 ± 0.071 |
| Cross-validated ROC-AUC | 0.986 ± 0.012 |
| Held-out test PR-AUC | 0.956 |
| Held-out test recall | 0.81 (caught 13/16 mules) |
| Held-out test precision | 0.87 (only 2 false positives in 1,801 legit) |
| Test F2 | 0.82 |
| Alert rate | 0.8% of accounts |
| Logistic baseline PR-AUC (sanity) | 0.389 → the large gap proves genuine non-linear signal, not leakage |
Why this is a strong result: 81% of mules caught at an 0.8% alert rate is a queue a real fraud team can work. The fused anomaly score was selected in every CV fold, addressing the "evolving fraud patterns" requirement. Every alert ships with SHAP reason codes.
See reports/model_report.md (model card), reports/feature_selection.md, and reports/data_profile.md for full detail.
Generated for the BOI Hackathon. Model decisions are explainable and auditable; capabilities demonstrated on the provided dataset, with a clear productionization roadmap to live regulatory/cross-channel feeds.