README.md

---
title: Scam Detector
emoji: 🛡️
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.9.0
app_file: app.py
pinned: false
license: mit
---

# 🛡️ Scam Detector

AI-powered SMS and URL scam detection, trained on India-specific fraud patterns. Flags messages and links as **Safe**, **Suspicious**, or **Scam** in real time.

**Live demo:** [huggingface.co/spaces/Sanjam19/Scam-Detector](https://huggingface.co/spaces/Sanjam19/Scam-Detector)

---

## What It Does

- **Text / SMS analysis** — detects phishing, bank impersonation, KYC fraud, digital arrest scams, OTP theft, investment fraud, and more
- **URL analysis** — ensemble classifier (3 models) + rule-based pattern matching to flag malicious links
- **User feedback loop** — every prediction collects thumbs up/down, stored for future retraining

---

## Model Architecture

### Text Classifier
- **Pipeline:** TF-IDF (5,000 features, unigrams + bigrams) → Logistic Regression (`class_weight='balanced'`)
- **Training data:** SMS Spam Collection (5,574 messages) + self-curated India-specific scam dataset (500 messages, 5× upweighted)
- **Threshold:** Scam ≥ 0.55 · Suspicious ≥ 0.35 · Safe < 0.35 — tuned to minimise false negatives given the asymmetric cost of missing a scam
- **Rule layer:** 35+ regex patterns covering social engineering tactics specific to Indian fraud (KYC scams, RBI impersonation, digital arrest, UPI suspension, SIM KYC, Aadhaar fraud)
- **Final score:** `max(ML score, rule score)` — conservative by design

### URL Ensemble (3 models, soft-voted)
| Model | Features | CV Balanced Accuracy |
|-------|----------|---------------------|
| TF-IDF char n-gram (3–5) + Logistic Regression | Raw URL string | 63.0% |
| Random Forest | 21 engineered features (hyphen count, TLD type, subdomain depth, brand+hyphen flag, etc.) | 71.2% |
| XGBoost | URL + red_flags + domain_pattern text (word TF-IDF) | 70.5% |

Soft vote averages probability vectors across all three. If any single model exceeds 85% confidence on MALICIOUS, it overrides the average. Final score blends ensemble (50%) with rule-based checks (50%).

---

## Performance

**Text model** (200-sample stratified test, SMS Spam Collection):

| Metric | Score |
|--------|-------|
| Accuracy | 98.5% |
| Recall | 100% |
| Precision | 97.1% |
| F1 | 98.5% |

Zero false negatives by design — the threshold is deliberately set to catch every scam at the cost of a small increase in false positives.

**URL ensemble** (3-fold CV on 250-label phishing URL dataset): 63–71% balanced accuracy across models. Ensemble + rules combination handles edge cases neither approach catches alone.

---

## India-Specific Scam Coverage

The self-curated dataset and rule layer cover fraud patterns underrepresented in standard SMS spam datasets:

- Bank impersonation (HDFC, SBI, ICICI, Axis, Kotak, RBI)
- KYC expiry / Aadhaar deactivation fraud
- Digital arrest scams (CBI, ED, cybercrime impersonation)
- UPI / wallet suspension threats
- SIM KYC fraud (Airtel, Jio, Vi)
- Investment fraud / pig butchering (crypto trading groups)
- Government scheme fraud (EPFO, GST, income tax refunds)
- Customs clearance fee scams
- Job registration fee fraud

---

## Why Not BERT?

Tried it. Zero-shot BERT got 32% accuracy — worse than a coin flip on this task. General language models don't understand spam-specific vocabulary patterns. TF-IDF trained directly on scam messages learns exactly the right features: `URGENT`, `claim`, `KYC`, `OTP`, `verify`, n-gram combinations that appear almost exclusively in scam messages. Simpler model, better data, better result.

---

## Feedback Loop

Every prediction shows a **Was this correct? Yes / No** prompt. Responses are stored in SQLite with timestamp, prediction, and confidence. This builds a labelled dataset of real-world predictions for future active learning retraining — the gap between model agreement rate and 100% identifies the hardest edge cases.

---

## File Structure

```
scam-detector/
├── app.py                          # Gradio UI + feedback DB
├── ml_utils.py                     # Text classifier + URL ensemble + rule layer
├── spam_model.pkl                  # Trained text model (auto-generated on first run)
├── url_ensemble.pkl                # Trained URL ensemble (auto-generated on first run)
├── scam_messages_complete_500.csv  # India-specific scam dataset
├── spam.csv                        # SMS Spam Collection (base training data)
├── scam_urls_training_250.csv      # Labelled URL dataset for ensemble training
├── evaluate.py                     # Evaluation script
├── requirements.txt
└── README.md
```

If `spam_model.pkl` or `url_ensemble.pkl` are absent, both models retrain automatically on startup using the CSV files above.

---

## Evaluation

```bash
python evaluate.py
```

Reproduces the 98.5% accuracy result on a 200-message stratified test split.

---

## Tech Stack

- **UI:** Gradio 6.9
- **ML:** scikit-learn, XGBoost
- **Text features:** TF-IDF (sklearn)
- **URL features:** engineered + char n-gram TF-IDF + word TF-IDF
- **Storage:** SQLite (feedback)
- **Deployment:** Hugging Face Spaces

---

## Contact

**Sanjam Das** — [sanjamdas2@gmail.com](mailto:sanjamdas2@gmail.com) · [LinkedIn](https://www.linkedin.com/in/sanjamdas/) · [GitHub](https://github.com/SD1920)