README.md

metadata

title: Scam Detector
emoji: 🛡️
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.9.0
app_file: app.py
pinned: false
license: mit

🛡️ Scam Detector

AI-powered SMS and URL scam detection, trained on India-specific fraud patterns. Flags messages and links as Safe, Suspicious, or Scam in real time.

Live demo: huggingface.co/spaces/Sanjam19/Scam-Detector

---

What It Does

• Text / SMS analysis — detects phishing, bank impersonation, KYC fraud, digital arrest scams, OTP theft, investment fraud, and more
• URL analysis — ensemble classifier (3 models) + rule-based pattern matching to flag malicious links
• User feedback loop — every prediction collects thumbs up/down, stored for future retraining

---

Model Architecture

Text Classifier

• Pipeline: TF-IDF (5,000 features, unigrams + bigrams) → Logistic Regression (class_weight='balanced')
• Training data: SMS Spam Collection (5,574 messages) + self-curated India-specific scam dataset (500 messages, 5× upweighted)
• Threshold: Scam ≥ 0.55 · Suspicious ≥ 0.35 · Safe < 0.35 — tuned to minimise false negatives given the asymmetric cost of missing a scam
• Rule layer: 35+ regex patterns covering social engineering tactics specific to Indian fraud (KYC scams, RBI impersonation, digital arrest, UPI suspension, SIM KYC, Aadhaar fraud)
• Final score: max(ML score, rule score) — conservative by design

URL Ensemble (3 models, soft-voted)

Model	Features	CV Balanced Accuracy
TF-IDF char n-gram (3–5) + Logistic Regression	Raw URL string	63.0%
Random Forest	21 engineered features (hyphen count, TLD type, subdomain depth, brand+hyphen flag, etc.)	71.2%
XGBoost	URL + red_flags + domain_pattern text (word TF-IDF)	70.5%

Soft vote averages probability vectors across all three. If any single model exceeds 85% confidence on MALICIOUS, it overrides the average. Final score blends ensemble (50%) with rule-based checks (50%).

---

Performance

Text model (200-sample stratified test, SMS Spam Collection):

Metric	Score
Accuracy	98.5%
Recall	100%
Precision	97.1%
F1	98.5%

Zero false negatives by design — the threshold is deliberately set to catch every scam at the cost of a small increase in false positives.

URL ensemble (3-fold CV on 250-label phishing URL dataset): 63–71% balanced accuracy across models. Ensemble + rules combination handles edge cases neither approach catches alone.

---

India-Specific Scam Coverage

The self-curated dataset and rule layer cover fraud patterns underrepresented in standard SMS spam datasets:

• Bank impersonation (HDFC, SBI, ICICI, Axis, Kotak, RBI)
• KYC expiry / Aadhaar deactivation fraud
• Digital arrest scams (CBI, ED, cybercrime impersonation)
• UPI / wallet suspension threats
• SIM KYC fraud (Airtel, Jio, Vi)
• Investment fraud / pig butchering (crypto trading groups)
• Government scheme fraud (EPFO, GST, income tax refunds)
• Customs clearance fee scams
• Job registration fee fraud

---

Why Not BERT?

Tried it. Zero-shot BERT got 32% accuracy — worse than a coin flip on this task. General language models don't understand spam-specific vocabulary patterns. TF-IDF trained directly on scam messages learns exactly the right features: URGENT, claim, KYC, OTP, verify, n-gram combinations that appear almost exclusively in scam messages. Simpler model, better data, better result.

---

Feedback Loop

Every prediction shows a Was this correct? Yes / No prompt. Responses are stored in SQLite with timestamp, prediction, and confidence. This builds a labelled dataset of real-world predictions for future active learning retraining — the gap between model agreement rate and 100% identifies the hardest edge cases.

---

File Structure

scam-detector/
├── app.py                          # Gradio UI + feedback DB
├── ml_utils.py                     # Text classifier + URL ensemble + rule layer
├── spam_model.pkl                  # Trained text model (auto-generated on first run)
├── url_ensemble.pkl                # Trained URL ensemble (auto-generated on first run)
├── scam_messages_complete_500.csv  # India-specific scam dataset
├── spam.csv                        # SMS Spam Collection (base training data)
├── scam_urls_training_250.csv      # Labelled URL dataset for ensemble training
├── evaluate.py                     # Evaluation script
├── requirements.txt
└── README.md

If spam_model.pkl or url_ensemble.pkl are absent, both models retrain automatically on startup using the CSV files above.

---

Evaluation

python evaluate.py

Reproduces the 98.5% accuracy result on a 200-message stratified test split.

---

Tech Stack

• UI: Gradio 6.9
• ML: scikit-learn, XGBoost
• Text features: TF-IDF (sklearn)
• URL features: engineered + char n-gram TF-IDF + word TF-IDF
• Storage: SQLite (feedback)
• Deployment: Hugging Face Spaces

---

Contact

Sanjam Das — sanjamdas2@gmail.com · LinkedIn · GitHub