IntegralBender/distilbert-spam-detector

DistilBERT Multilingual Spam & Phishing Detector

DistilBERT Multilingual Spam & Phishing Detector is a high-performance, fine-tuned model engineered for real-time detection of spam, phishing emails and unsolicited malicious messages across multiple languages.

Built via an automated MLOps pipeline, this model merges general spam sources with phishing benchmarks to achieve robust generalization and low inference overhead.

An INT8 Quantized ONNX variant optimized for low-latency CPU and edge deployments is available at IntegralBender/distilbert-spam-detector-onnx.

Table of Contents

1. Model Overview
2. Intended Uses & Limitations
3. Usage & Integration
4. Dataset & Preprocessing Pipeline
5. Training & Evaluation
6. Framework Versions & Environment

Model Overview

• Architecture: DistilBERTForSequenceClassification
• Base Model: distilbert/distilbert-base-multilingual-cased
• Primary Task: Binary Text Classification (HAM: 0, SPAM: 1)
• Max Sequence Length: 512 tokens
• License: Apache 2.0
• Supported Languages: Multilingual (Supports 20+ primary training languages and inherits base cased vocabulary)

Intended Uses & Limitations

Primary Use Cases

• Inbox Security: Real-time filtering and pre-screening of incoming email payloads.
• Content Moderation: Automated detection of spam in comments, chat applications and online form submissions.
• Phishing Defense: Flagging unauthorized link-injection attacks and social engineering text patterns.

Out-of-Scope & Limitations

• Decision Authority: Should be utilized as an automated flagging system rather than a sole arbiter for permanent email deletion.
• Slang & Obfuscation: While highly effective against standard spam vectors, heavy leetspeak or unusual character obfuscations may occasionally lead to false negatives.

Usage & Integration

Because the model was trained with native id2label mappings, prediction outputs are returned directly as HAM or SPAM.

Standard Inference (PyTorch)

from transformers import pipeline

# Load pipeline directly from Hugging Face Hub
classifier = pipeline(
    "text-classification", 
    model="IntegralBender/distilbert-spam-detector"
)

# Test with multilingual inputs
samples = [
    "Verify your bank account immediately by clicking this secure link: http://bit.ly/fake-link",
    "Lütfen yarınki toplantı notlarını sisteme yüklemeyi unutmayın.",
    "WICHTIG: Ihr Konto wurde vorübergehend gesperrt. Bitte aktualisieren Sie Ihre Daten hier: http://bank-sicherheit-de.com",
    "Bonjour, la réunion de demain est décalée à 14h. Merci de confirmer votre présence."
]

results = classifier(samples)

for text, res in zip(samples, results):
    print(f"[{res['label']}] (Confidence: {res['score']:.4f}) -> {text}")

Dataset & Preprocessing Pipeline

The training corpus was dynamically aggregated, standardized, concatenated and globally shuffled from five open-source datasets to ensure robust class distribution:

• M-Arjun/SpamShield-Datasets
• SetFit/enron_spam
• ucirvine/sms_spam
• Deysi/spam-detection-dataset
• zefang-liu/phishing-email-dataset

Preprocessing & Data Cleaning

• Validation: Rows containing None values or empty string payloads are strictly filtered out during streaming.
• Label Normalization & Feature Alignment: Labels are dynamically matched against a defined set of indicators and cast to binary int64 (0 for HAM, 1 for SPAM). Features across all five datasets are strictly mapped to uniform text and label columns to prevent schema conflicts.
• Strict Deduplication & Leakage Prevention: After concatenation, the unified corpus undergoes a fast, index-based deduplication process on the text payloads. This critical step removes overlapping entries across the different dataset sources, effectively preventing data leakage between the train and test sets and mitigating overfitting.
• Data Partitioning & Reproducibility: The deduplicated dataset is globally shuffled and split into train/test partitions. A fixed initialization seed (seed=42) is rigorously enforced across the pipeline to guarantee exact reproducibility of the dataset state and evaluation metrics for anyone retraining from source.

Training & Evaluation

Training Configuration & Hardware

• Hardware Requirements: The pipeline includes a strict hardware check, requiring a dedicated CUDA-enabled GPU for acceleration.
• Optimization: Trained using native automatic mixed precision (FP16) on CUDA, orchestrated through Hydra configuration files.

Training Hyperparameters

The model was fine-tuned using the following configuration:

• Learning Rate: 2e-05
• Train Batch Size: 32
• Eval Batch Size: 64
• Seed: 42
• Optimizer: AdamW Fused (adamw_torch_fused) with betas=(0.9, 0.999) and epsilon=1e-08
• Learning Rate Scheduler: Linear
• Number of Epochs: 3
• Mixed Precision: Native AMP (FP16)

Training Progression & Results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Precision	Recall
0.0245	1.0	5673	0.0553	0.9843	0.9839	0.9932	0.9748
0.0169	2.0	11346	0.0374	0.9901	0.99	0.9905	0.9894
0.0077	3.0	17019	0.0434	0.9916	0.9915	0.9932	0.9897

Framework Versions & Environment

The model pipeline was built and validated within the following software environment:

• Transformers: 4.57.6
• PyTorch: 2.11.0+cu130
• Datasets: 4.8.5
• Tokenizers: 0.22.2