project_problem.md

Project 10 — Autoencoder Anomaly Detection

Level: Advanced | Dataset: Credit Card Fraud (Kaggle) | Framework: PyTorch

---

Objective

Build an Autoencoder that learns the distribution of normal transactions and flags anomalies via high reconstruction error. Cover: encoder-decoder architecture, bottleneck, reconstruction loss, threshold tuning, unsupervised anomaly detection.

---

Project Structure

10_autoencoder_anomaly/
├── notebooks/
│   ├── 01_eda.ipynb
│   ├── 02_preprocessing.ipynb
│   └── 03_train_evaluate.ipynb
├── data/raw/creditcard.csv
├── data/processed/
├── models/model.pkl
├── charts/
├── path_utils.py
├── dashboard_core.py
└── app.py

Dataset: mlg-ulb/credit-card-fraud-detection — Kaggle. 284,807 transactions, 492 fraud (0.17% fraud rate). Features: V1-V28 (PCA-transformed), Amount, Time. Target: Class (0=normal, 1=fraud).

---

Notebook 01 — EDA (01_eda.ipynb)

STOP 1 — Load & Class Distribution

• Load creditcard.csv
• Print class distribution: 284,315 normal, 492 fraud
• This is extreme imbalance: 99.83% vs 0.17%
• Agent stops here. Explain:
  • Why this is a perfect Autoencoder use case: we have very few fraud examples
  • The unsupervised insight: train ONLY on normal → AE learns to reconstruct normal well → fraud reconstructed poorly
  • Why supervised models struggle here: too few fraud examples even with class weights
  • Real-world context: in production, fraud patterns change constantly — unsupervised is more robust
• Wait for user confirmation before continuing

STOP 2 — Feature Analysis

• Plot distribution of Amount — heavily skewed, log transform
• Plot distribution of V1, V5, V14 (most fraud-discriminative PCA components)
• Overlay normal vs fraud distributions for V14
• Agent stops here. Explain:
  • What V1-V28 are: principal components of original transaction features (anonymized by Kaggle)
  • Why Amount needs special treatment (raw dollar amount vs scaled PCA features)
  • What overlapping distributions mean: fraud and normal transactions look similar to linear models
  • How the AE exploits the subtle difference: it learns the joint distribution of all features
• Wait for confirmation

STOP 3 — Reconstruction Concept Walkthrough

• Explain (with markdown cells) what reconstruction means:
  • AE takes input x → compresses to z (bottleneck) → reconstructs x̂
  • Loss: MSE(x, x̂) averaged over all features
  • At inference: high MSE = anomalous
• Agent stops here. Explain:
  • The information bottleneck principle: compression forces the model to learn the "essence"
  • Why bottleneck width matters: too wide = AE memorizes everything (no anomaly detection), too narrow = loses normal patterns too
  • What reconstruction error distribution looks like for normal vs fraud
• Wait for confirmation

---

Notebook 02 — Preprocessing (02_preprocessing.ipynb)

STOP 4 — Isolation of Normal Class

• Separate: normal_df = df[df['Class'] == 0]
• Training set: 80% of normal only (no fraud in training)
• Validation set: 10% normal + ALL 492 fraud (to tune threshold)
• Test set: remaining 10% normal + reserved 100 fraud
• Agent stops here. Explain:
  • Why we train ONLY on normal data — this is the core principle of AE-based anomaly detection
  • Why we include fraud in validation: to find the optimal reconstruction error threshold
  • The correct split strategy for unsupervised anomaly detection
• Wait for confirmation

STOP 5 — Feature Scaling

• Log transform Amount: np.log1p(df['Amount'])
• Drop Time column (not informative after PCA)
• StandardScaler fit on normal train features only
• Apply to normal train, val (normal+fraud), test (normal+fraud)
• Agent stops here. Explain:
  • Why log1p for Amount: log(1+x) handles zero correctly, compresses large values
  • Why StandardScaler fit only on normal train: we're assuming normal distribution of normal transactions
  • What happens if we scale fraud using fraud statistics (leakage, defeats the purpose)
• Wait for confirmation

STOP 6 — Tensor Dataset

• Normal train: X only (no labels needed for training — unsupervised)
• Val/Test: (X, y) pairs where y is the fraud label for evaluation
• DataLoader for train: batch_size=256, shuffle=True
• Agent stops here. Explain:
  • Why training DataLoader has no labels: AE is trained to minimize reconstruction error, not classify
  • How this is fundamentally different from all previous supervised projects
  • What "unsupervised learning" means in practice
• Wait for confirmation

---

Notebook 03 — Train & Evaluate (03_train_evaluate.ipynb)

STOP 7 — Autoencoder Architecture

Encoder:
  Linear(29, 64) → ReLU → Dropout(0.1)
  Linear(64, 32) → ReLU
  Linear(32, 16) → ReLU  [bottleneck = 16]

Decoder:
  Linear(16, 32) → ReLU
  Linear(32, 64) → ReLU
  Linear(64, 29)          [no activation — reconstruct any value]

Forward: x → z = encode(x) → x_hat = decode(z) → return x_hat

• Agent stops here. Explain:
  • Symmetric encoder-decoder: decoder mirrors encoder structure
  • Bottleneck dimension=16: compresses 29 features to 16 (forced information bottleneck)
  • Why no activation at decoder output: output must match input range (any real value after scaling)
  • What the latent space z represents: compressed representation of the transaction
  • How to choose bottleneck size: experiment — too small loses normal patterns, too large = no compression
• Wait for confirmation

STOP 8 — Reconstruction Loss

• Use nn.MSELoss(reduction='none') — keep per-sample, per-feature losses
• Average over features for per-sample reconstruction error
• Training loss: mean of per-sample errors
• Agent stops here. Explain:
  • Why reduction='none': we need per-sample error at inference time
  • What reconstruction error for ONE sample looks like: scalar value (mean over 29 features)
  • Why MSE penalizes large reconstruction errors quadratically — good for detecting anomalies
  • Alternative: MAE loss — less sensitive to outliers (sometimes better for AE)
• Wait for confirmation

STOP 9 — Training Loop

• Train on NORMAL ONLY for 50 epochs
• Track train reconstruction error per epoch
• Also compute val reconstruction error for normal vs fraud separately
• Plot: normal reconstruction error distribution vs fraud reconstruction error distribution
• Agent stops here. Explain:
  • What we expect to see: two distributions, fraud shifted right (higher error)
  • Why the distributions might overlap: some fraud looks like normal, some normal looks weird
  • The separation quality directly predicts AUC
  • What "collapse" looks like if bottleneck is too wide: both distributions identical
• Wait for confirmation

STOP 10 — Threshold Tuning

• Compute reconstruction error for ALL validation samples (normal + fraud)
• Try thresholds from min to max error at 100 steps
• For each threshold: compute Precision, Recall, F1
• Plot F1 vs threshold curve
• Select threshold that maximizes F1 (or recall, depending on business requirement)
• Agent stops here. Explain:
  • What threshold selection is: converting a continuous score to binary prediction
  • The precision-recall tradeoff at different thresholds
  • In fraud detection, what is worse: false positive (block good transaction) vs false negative (miss fraud)?
  • Why we tune on val, evaluate on test (never touch test during tuning)
• Wait for confirmation

STOP 11 — Evaluation on Test Set

• Apply tuned threshold to test set
• Compute: Precision, Recall, F1, AUC-ROC, AUC-PR
• Plot ROC curve and Precision-Recall curve
• Agent stops here. Explain:
  • Why AUC-PR is more informative than AUC-ROC for extreme imbalance
  • What AUC-PR = 0.5 means on a 0.17% fraud rate (baseline = 0.0017!)
  • Why ROC can be misleadingly optimistic with extreme imbalance
  • The business metric: catch rate (recall on fraud) at a given false positive rate
• Wait for confirmation

STOP 12 — Latent Space Visualization

• Encode all test samples (normal + fraud) to get z vectors [N, 16]
• Apply t-SNE or PCA to reduce to 2D
• Plot with color: blue=normal, red=fraud
• Agent stops here. Explain:
  • What we hope to see: fraud forming clusters away from normal
  • What t-SNE shows that PCA doesn't: non-linear clustering structure
  • Why fraud might not perfectly separate in latent space (some fraud IS similar to normal transactions)
  • How this visualization helps in understanding model failure modes
• Wait for confirmation

STOP 13 — Save & Inference

• Save model.state_dict(), scaler, threshold
• Write predict_fraud(transaction_dict) → label, reconstruction_error, is_fraud
• Agent stops here. Explain:
  • Complete inference pipeline: dict → preprocess (log Amount, scale) → tensor → model.eval() → reconstruct → MSE → compare to threshold → return
  • Why we save the threshold with the model (it's part of the "model")
  • How to update threshold in production as fraud patterns evolve
• Wait for confirmation

---

dashboard_core.py

Functions:

• load_model_scaler_threshold() → model, scaler, threshold
• predict_fraud(transaction_dict) → reconstruction_error, is_fraud, bool
• get_error_distributions() → (normal_errors, fraud_errors) arrays
• get_roc_pr_curves() → dict of curve data
• get_latent_viz() → 2D coords + labels

---

app.py — Streamlit (~80 lines)

Sections:

1. Sidebar: sliders for V1, V14, V17, Amount (most discriminative features)
2. Main: "Analyze Transaction" → show reconstruction error + fraud/normal verdict
3. Tab 1: Training reconstruction error curve
4. Tab 2: Error distribution histogram (normal vs fraud overlap)
5. Tab 3: ROC + PR curves

---

Key Concepts Covered

• Autoencoder architecture (encoder, bottleneck, decoder)
• Information bottleneck principle
• Training on normal only (unsupervised anomaly detection)
• Reconstruction loss (MSE reduction='none' for per-sample)
• Threshold tuning on validation set
• AUC-PR vs AUC-ROC for imbalanced data
• Latent space visualization with t-SNE
• Full unsupervised learning pipeline