docs/MODELING.md

RazorShield — Modeling, Calibration & False-Positive Reduction Documentation

This document describes the modeling methodology, probability calibration, deployable feature isolation, threshold tuning, hard-negative failure investigation, and cost-sensitivity benchmarks for the RazorShield defensive AI fraud-spike detection system.

Disclaimer: This document represents offline model training and evaluation benchmarking (Phases 3 & 4). It does NOT constitute or claim full production readiness. Further risk engine integration, latency profiling, and real-time streaming validation are required in subsequent phases.

---

1. Probability Calibration (Dataset A Transaction Model)

Why Raw Probability Calibration Was Needed

Raw XGBoost probabilities trained on imbalanced datasets using scale_pos_weight = 27.5 suffer from severe probability distortion. Raw output scores are shifted upwards, resulting in a high Brier Score (0.0989) and an Expected Calibration Error (ECE) of 21.85%. Calibration maps model confidence scores to true empirical probabilities $P(\text{fraud} \mid \text{txn})$.

Calibration Methods Evaluated (Fitted Strictly on Validation Data)

1. Raw XGBoost: Uncalibrated predictions.
2. Sigmoid Calibration (Platt Scaling): Logistic regression fitted on validation prediction logits.
3. Isotonic Calibration: Non-parametric isotonic regression fitted on validation prediction probabilities.

Calibration Benchmarks

Method	Validation Brier Score	Validation Log Loss	Validation ECE	Test Brier Score	Test Log Loss	Test ECE
Raw XGBoost	0.098948	0.342828	21.851%	0.104293	0.356667	22.567%
Sigmoid (Platt)	0.029633	0.121364	0.210%	0.030664	0.126617	0.353%
Isotonic (Selected)	0.029433	0.120360	0.000%	0.030629	0.126998	0.188%

Selection: Isotonic Calibration achieved the minimum Validation Brier score (0.029433) and reduced Expected Calibration Error from 21.85% to 0.188% on the Test set.

---

2. Dataset B — Hard Negative Investigation & Feature Improvements

Volume-Only Hard Negative Failure Analysis

In Phase 3, the deployable spike detector exhibited a 39.35% false-alert rate on volume_only_spike scenarios (e.g. flash sales, promotional campaigns).

Root Cause

Flash sales generate high transaction volume ($\approx 4.6\text{x}$ baseline). In Phase 3, the model relied heavily on velocity_ratio ($\text{rolling_txn_15m} / \text{baseline_txn_15m}$). Because raw transaction volume surged, the detector triggered false fraud-spike alerts even though the underlying transaction fraud rate remained at baseline (~0.8%).

New Deployable Fraud-Excess Features (Phase 4)

To decouple legitimate volume surges from genuine fraud surges, 7 new deployable features were engineered:

1. fraud_signal_ratio: $\text{estimated_fraud_rate_15m} / \text{baseline_fraud_rate}$
2. estimated_fraud_count_15m: Sum of transaction calibrated fraud probabilities $\sum \hat{p}_i$ in the 15-minute window.
3. expected_fraud_count_15m: $\text{baseline_fraud_rate} \times \text{rolling_txn_15m}$
4. fraud_excess_ratio: $\text{estimated_fraud_count_15m} / \text{expected_fraud_count_15m}$
5. volume_deviation: $\text{rolling_txn_15m} / \text{baseline_txn_15m}$
6. fraud_excess_minus_velocity: $\text{fraud_excess_ratio} - \text{velocity_ratio}$
7. amount_shift_indicator: $\text{amount} / \text{baseline_amount}$

Why Fraud-Excess Disambiguates Flash Sales

• Flash Sales (volume_only_spike): Both actual volume and expected fraud count increase proportionally. Thus, $\text{fraud_excess_ratio} \approx 1.0$ and $\text{fraud_excess_minus_velocity} < 0$, preventing false alerts.
• Genuine Fraud Spikes (fraud_spike): Calibrated transaction fraud probabilities surge. Thus, $\text{fraud_excess_ratio} \gg 1.0$ ($\approx 16.6\text{x}$) and $\text{fraud_excess_minus_velocity} \gg 0$, triggering valid alerts.

---

3. Cost-Sensitive Threshold Optimization Methodology

Threshold optimization is performed strictly on the Validation set by minimizing expected financial loss across illustrative cost ratios $C_{\text{FN}} : C_{\text{FP}}$ (where $C_{\text{FP}} = 1.0$):

$$\text{Expected Cost}=(C_{\text{FP}}\times \text{FP})+(C_{\text{FN}}\times \text{FN})\backslash text\{Expected\; Cost\}\; =\; (C\_\{\backslash text\{FP\}\}\; \backslash times\; \backslash text\{FP\})\; +\; (C\_\{\backslash text\{FN\}\}\; \backslash times\; \backslash text\{FN\})$$

Selected thresholds are frozen and evaluated once on the Test set.

Cost-Optimized Threshold Results (Dataset B Spike Detector)

Cost Ratio ($C_{\text{FN}} : C_{\text{FP}}$)	Selected Val Threshold	Val FP	Val FN	Val Expected Cost	Test FP	Test FN	Test Precision	Test Recall	Test Expected Cost
5 : 1	0.04	5,433	181	$6,338.00	3,194	150	0.4104	0.9368	$3,944.00
10 : 1	0.04	5,433	181	$7,243.00	3,194	150	0.4104	0.9368	$4,694.00
20 : 1	0.02	6,679	91	$8,499.00	3,981	0	0.3735	1.0000	$3,981.00
50 : 1	0.02	6,679	91	$11,229.00	3,981	0	0.3735	1.0000	$3,981.00

---

4. Phase 3 vs Phase 4 Comparison Table

Metric / Scenario	Phase 3 (Baseline)	Phase 4 (Calibrated + Fraud-Excess Features)	Improvement / Difference
Transaction Model ECE	21.85%	0.188%	-21.66% ECE (Calibrated)
Transaction Model Brier Score	0.0989	0.0294	-0.0695 Brier Score
volume_only_spike False Alert Rate (at $T=0.30$)	39.35%	5.27%	-34.08% False Alert Reduction
amount_shift False Alert Rate	1.21%	0.00%	-1.21% False Alert Reduction
normal False Alert Rate	0.39%	0.00%	-0.39% False Alert Reduction
Fraud Spike Precision (at $T=0.30$)	48.51%	68.24%	+19.73% Precision
Spike Detector ROC-AUC	0.8672	0.9396	+0.0724 ROC-AUC

---

5. Remaining Limitations

1. Trade-off between False Alerts & Early Detection: Tuning the threshold to $T=0.30$ reduces volume_only_spike false alerts to 5.27%, but catches fraud spikes during active high-confidence windows.
2. Merchant Campaign Registration: Automated detection benefits significantly if merchants register scheduled flash sale windows in advance via API to suppress velocity-triggered warnings.