# Deep Learning Project: Spam Detection using Transformers **Course**: Deep Learning with Python (2025) **Instructor**: Benoit Mialet **Topic**: NLP - Text Classification (Spam vs Ham) **Model**: DistilBERT (PyTorch / Hugging Face) --- ## 1. Introduction ### 1.1 What & Why The objective of this project is to develop a robust deep learning model for classifying emails as either "spam" or "ham" (legitimate). Email filtering is a critical application of Natural Language Processing (NLP) that helps improve user experience and security by automatically identifying unsolicited or malicious content. ### 1.2 Task Selection We chose the **Text Classification** task, specifically binary classification. This task is well-suited for demonstrating the power of Transfer Learning and Transformer architectures in understanding the nuances of human language. ### 1.3 Relevance Spam detection remains a relevant challenge as spamming techniques evolve. Traditional rule-based systems often fail to capture the semantic meaning of messages. Deep learning models, particularly Transformers, can capture long-range dependencies and contextual information, leading to higher accuracy and better generalization. ### 1.4 State of the Art Modern NLP has been revolutionized by the Transformer architecture (Vaswani et al., 2017). Models like BERT (Bidirectional Encoder Representations from Transformers) and its variants (DistilBERT, RoBERTa) have set new benchmarks in text classification by pre-training on large corpora and fine-tuning on specific tasks. --- ## 2. Method ### 2.1 Overall Strategy Our strategy involves: 1. **Exploratory Data Analysis (EDA)** to understand the dataset characteristics. 2. **Data Preprocessing** including tokenization and padding. 3. **Fine-tuning a Pre-trained Model** (DistilBERT) using the Hugging Face `transformers` library and PyTorch. 4. **Rigorous Evaluation** using metrics like Accuracy, Precision, Recall, and F1-score. ### 2.2 Dataset Description & EDA The dataset used is `mail_data.csv`, containing 5,572 messages labeled as 'ham' or 'spam'. - **Total Samples**: 5,572 - **Ham**: 4,825 (86.6%) - **Spam**: 747 (13.4%) - **Imbalance**: The dataset is significantly imbalanced, which we addressed by using stratified splitting and monitoring the F1-score. **EDA Findings**: - Spam messages tend to be longer on average than ham messages. - Common keywords in spam include "free", "win", "winner", "call", "claim". - Ham messages are more conversational and vary greatly in length. ### 2.3 Data Preprocessing - **Tokenization**: We used the `DistilBertTokenizer` to convert raw text into input IDs and attention masks. - **Truncation & Padding**: All sequences were padded or truncated to a maximum length of 128 tokens to ensure uniform input size for the model. - **Train/Test Split**: 80% training (4,457 samples) and 20% testing (1,115 samples), with stratification to maintain class proportions. ### 2.4 Model Architecture We utilized **DistilBERT** (`distilbert-base-uncased`), a smaller, faster, and lighter version of BERT that retains 97% of its performance. It has 6 layers, 768 hidden units, and 12 attention heads, totaling approximately 66 million parameters. ### 2.5 Training Setup - **Optimizer**: AdamW with a learning rate of 2e-5. - **Scheduler**: Linear warmup for 500 steps. - **Loss Function**: Cross-Entropy Loss. - **Batch Size**: 16 for training, 64 for evaluation. - **Epochs**: 3 (stopped early after 1 epoch due to high performance and resource constraints). - **Hardware**: CPU (simulated environment). --- ## 3. Results ### 3.1 Performance Metrics The model achieved exceptional results after just one epoch of fine-tuning: | Metric | Value | | :--- | :--- | | **Accuracy** | 99.10% | | **Precision (Spam)** | 98.60% | | **Recall (Spam)** | 94.63% | | **F1-Score (Spam)** | 96.58% | ### 3.2 Confusion Matrix | | Predicted Ham | Predicted Spam | | :--- | :---: | :---: | | **Actual Ham** | 964 | 2 | | **Actual Spam** | 8 | 141 | The model correctly identified 141 out of 149 spam messages while only misclassifying 2 legitimate messages as spam (False Positives). --- ## 4. Discussion ### 4.1 Interpretation The high accuracy and F1-score indicate that DistilBERT is highly effective for this task. The model successfully learned the semantic patterns that distinguish spam from ham, even with a relatively small and imbalanced dataset. ### 4.2 What Worked - **Transfer Learning**: Using a pre-trained model allowed us to achieve near-perfect results with minimal training time. - **Hugging Face Trainer**: Simplified the training loop and handled evaluation efficiently. - **Tokenization**: The subword tokenization of BERT handles out-of-vocabulary words better than traditional word-based methods. ### 4.3 Limitations - **Dataset Size**: While sufficient for this project, a larger and more diverse dataset would be needed for a production-grade system. - **Class Imbalance**: Although the model performed well, the recall for spam (94.63%) is slightly lower than for ham, reflecting the imbalance. - **Adversarial Attacks**: Sophisticated spam might use techniques to bypass Transformer-based filters, which was not explored here. ### 4.4 Future Improvements - **Data Augmentation**: Techniques like back-translation could help balance the dataset. - **Hyperparameter Tuning**: Exploring different learning rates and batch sizes. - **Deployment**: Creating a Gradio interface on Hugging Face Spaces for real-time testing. - **Model Compression**: Quantization or pruning to make the model even lighter for mobile deployment. --- ## 5. Conclusion This project successfully demonstrated the application of Deep Learning for spam detection. By leveraging the DistilBERT architecture and the Hugging Face ecosystem, we built a model that achieves over 99% accuracy. The results highlight the efficiency of transfer learning in NLP, proving that even with limited resources, state-of-the-art performance is attainable. --- ## 6. References 1. Vaswani, A., et al. (2017). "Attention Is All You Need." 2. Sanh, V., et al. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter." 3. Wolf, T., et al. (2020). "Transformers: State-of-the-Art Natural Language Processing."