Spaces:
Runtime error
Runtime error
| title: MuleGuard | |
| emoji: 🛡️ | |
| colorFrom: blue | |
| colorTo: green | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: true | |
| license: mit | |
| short_description: Real-time AI/ML detection of suspicious mule accounts | |
| # MuleGuard — AI/ML Detection of Suspicious Mule Accounts | |
| > **BOI Hackathon · Problem Statement 2** — *AI/ML-Based Classification of Suspicious Mule Accounts* | |
| > | |
| > An end-to-end system that ingests account/transaction features, scores every account for mule-risk in real time, explains every alert, and streams alerts to a fraud-analyst console — directly targeting the circulation of fraudulent proceeds through mule accounts. | |
| --- | |
| ## 1. Problem | |
| Banks face growing cyber-enabled fraud where **mule accounts** receive, transfer, and conceal fraudulent funds across channels. Rule-based monitoring misses evolving patterns. We must **learn behavioral & transactional patterns** from the provided data to **classify suspicious/mule accounts**, with anomaly detection, predictive risk scoring, and intelligent alerts. | |
| **Dataset:** `data/DataSet.csv` — **9,082 accounts × 3,924 anonymized features** (`F1`–`F3924`). **Target = `F3924`**. Severe imbalance: **9,001 legit (0) vs 81 mule (1) → 0.89% positive**. 18 features flagged by the bank as high-signal. | |
| ## 2. Our Solution — "MuleGuard" | |
| A supervised **gradient-boosted classifier fused with an unsupervised anomaly score**, wrapped in a real-time scoring service, a live feed simulator, and an analyst dashboard: | |
| ``` | |
| DataSet.csv ─▶ Feature Pipeline ─▶ Model + Calibrator + SHAP ─┐ | |
| ├─▶ artifacts/ | |
| Feed Simulator ─(streams accounts/tickets)─▶ FastAPI /score ───┘ | |
| │ risk 0-100, tier, reason codes, alert | |
| ▼ | |
| Streamlit Analyst Console | |
| ``` | |
| **Why this wins:** proven imbalanced-tabular method + novelty detection (catches *evolving* fraud) + full explainability (auditable alerts) + a working real-time demo that maps to the regulatory-feed vision. | |
| ## 3. Architecture & Engineering Approach | |
| - **Anti-leakage discipline** — all transforms fit inside CV folds / train only. | |
| - **Honest metrics** — PR-AUC, ROC-AUC, recall@precision, F2 (never accuracy). | |
| - **Repeated stratified CV** — robust estimates despite only 81 positives. | |
| - **Threshold tuning + probability calibration** — meaningful 0–100 risk scores. | |
| - **SHAP reason codes** — every alert is explainable. | |
| Built using a custom `.claude/` toolkit (agents + skills) authored **before** implementation — see §6. | |
| ## 4. Repository Layout | |
| ``` | |
| .claude/ custom agents + skills (incl. vendored superpowers & get-shit-done) | |
| data/ DataSet.csv (gitignored) | |
| src/data/ load, profile, clean | |
| src/features/ selection + engineering pipeline | |
| src/models/ train, evaluate, calibrate, explain | |
| src/api/ FastAPI scoring service | |
| src/simulator/ real-time feed simulator | |
| dashboard/ Streamlit analyst console | |
| artifacts/ trained model, calibrator, threshold, feature list, SHAP explainer | |
| reports/ data profile, feature selection, model card, figures | |
| docs/ architecture diagrams + spec | |
| deck/ pitch deck (.pptx) | |
| ``` | |
| ## 5. How to Run | |
| ```bash | |
| python3 -m venv .venv && .venv/bin/pip install -r requirements.txt | |
| .venv/bin/python -m src.data.profile # 1. profile data -> reports/data_profile.md | |
| .venv/bin/python -m src.features.build # 2. build features -> artifacts/ | |
| .venv/bin/python -m src.models.train # 3. train + evaluate -> artifacts/, reports/model_report.md | |
| .venv/bin/python -m src.models.explain # 4. SHAP explainability -> artifacts/, reports/figures/ | |
| .venv/bin/uvicorn src.api.main:app --port 8000 # 5. scoring API | |
| .venv/bin/python -m src.simulator.feed # 6. stream live alerts | |
| .venv/bin/streamlit run dashboard/app.py # 7. analyst console | |
| ``` | |
| ## 6. Custom `.claude/` Toolkit (built first, then used) | |
| **Agents** (`.claude/agents/`): `data-profiler`, `feature-engineer`, `ml-trainer`, `explainability-analyst`, `fraud-domain-expert`, `demo-builder`. | |
| **Skills** (`.claude/skills/`): `imbalanced-fraud-classification`, `fraud-feature-engineering` (authored), plus **`superpowers`** and **`get-shit-done`** vendored from GitHub as-is. | |
| --- | |
| ## 7. Progress Tracker | |
| | # | Phase | Status | | |
| |---|-------|--------| | |
| | 0 | Scaffold: structure, venv, deps, `.claude/` agents & skills | ✅ Done | | |
| | 1 | Data profiling (`reports/data_profile.md`) | ✅ Done | | |
| | 2 | Feature selection + engineering pipeline (103 features) | ✅ Done | | |
| | 3 | Model training, CV, calibration, threshold | ✅ Done | | |
| | 4 | SHAP explainability | ✅ Done | | |
| | 5 | FastAPI scoring service | ✅ Done | | |
| | 6 | Real-time feed simulator | ✅ Done | | |
| | 7 | Streamlit analyst dashboard | ✅ Done | | |
| | 8 | Architecture diagrams (`docs/`) | ✅ Done | | |
| | 9 | Pitch deck (`deck/MuleGuard_Pitch.pptx`) | ✅ Done | | |
| ## 8. Results | |
| **Model:** Calibrated LightGBM fused with an Isolation Forest anomaly score · **103 features** selected from 3,924. | |
| ### Leakage defense (the rigor that makes the numbers trustworthy) | |
| Two label-leaking features were detected and **removed** — without this, the model "achieved" a meaningless 100%: | |
| - **`F3912`** — a binary fraud-flag aligned ~perfectly with the label (single-feature AUC **0.987**). | |
| - **`F2230`** — observation month; all legit accounts in Oct-25, all mules in Sep/Nov/Dec-25 (temporal artifact, AUC **1.000**). | |
| ### Headline metrics | |
| | Metric | Value | | |
| |---|---| | |
| | **Cross-validated PR-AUC** (repeated stratified) | **0.882 ± 0.071** | | |
| | Cross-validated ROC-AUC | 0.986 ± 0.012 | | |
| | Held-out test PR-AUC | 0.956 | | |
| | Held-out test **recall** | **0.81** (caught 13/16 mules) | | |
| | Held-out test **precision** | **0.87** (only 2 false positives in 1,801 legit) | | |
| | Test F2 | 0.82 | | |
| | Alert rate | **0.8%** of accounts | | |
| | Logistic baseline PR-AUC (sanity) | 0.389 → the large gap proves genuine non-linear signal, **not** leakage | | |
| **Why this is a strong result:** 81% of mules caught at an **0.8% alert rate** is a queue a real fraud team can work. The fused anomaly score was selected in **every** CV fold, addressing the "evolving fraud patterns" requirement. Every alert ships with SHAP reason codes. | |
| See `reports/model_report.md` (model card), `reports/feature_selection.md`, and `reports/data_profile.md` for full detail. | |
| --- | |
| *Generated for the BOI Hackathon. Model decisions are explainable and auditable; capabilities demonstrated on the provided dataset, with a clear productionization roadmap to live regulatory/cross-channel feeds.* | |