Abdallah-Thuieb/distilbert-spam-classification-binary

DistilBERT Spam Classification (Binary)

An end-to-end project for spam vs ham text classification, including dataset cleaning, classical ML baselines (TF‑IDF + linear models), and Transformer fine‑tuning using distilbert-base-uncased [file:1].

Task

• Type: Text classification [file:1]
• Labels: ham (0), spam (1) [file:1]

Dataset

• Main columns:
  • Content (training text) [file:1]
  • SpamHam (label: spam/ham, with some missing NaN entries) [file:1]
• Dataset snapshot shown in the notebook:
  • Total rows: 2241 [file:1]
  • spam: 830, ham: 776, NaN: 635 [file:1]
• Extra feature engineering:
  • URLSource derived from URL patterns (twitter / youtube / other) [file:1]

Preprocessing

• Fill missing Content values using TweetText when available [file:1].
• Remove unused columns (e.g., Time, Date2, Date, Author, URL, TweetText) [file:1].
• Stratified split:
  • Test split: 15% [file:1]
  • Validation split: 15% from the remaining train/val set [file:1]

Models

Classical ML Baselines

• Features:
  • Word TF‑IDF n‑grams: (1, 2) [file:1]
  • Character TF‑IDF n‑grams: (3, 5) [file:1]
  • Combined with FeatureUnion [file:1]
• Algorithms:
  • Logistic Regression (class_weight="balanced") [file:1]
  • LinearSVC (class_weight="balanced") [file:1]

Transformer Fine-Tuning (DistilBERT)

• Base model: distilbert-base-uncased [file:1]
• Tokenization:
  • MAXLEN = 128 [file:1]
• Training (Trainer):
  • Learning rate: 2e-5 [file:1]
  • Train batch size: 16 [file:1]
  • Eval batch size: 32 [file:1]
  • Epochs: 5 [file:1]
  • Best model selected by F1 [file:1]

Results

Logistic Regression (TF‑IDF)

• Validation accuracy: ~0.971 [file:1]
• Test accuracy: ~0.946 [file:1]

LinearSVC (TF‑IDF)

• Validation accuracy: ~0.967 [file:1]
• Test accuracy: ~0.971 [file:1]

How to Run

pip install -q --upgrade transformers accelerate datasets ipywidgets tqdm scikit-learn