Create README.md

README.md

@@ -0,0 +1,60 @@
+# **Toxic Comment Classification with Transformer Optimization**
+
+This project demonstrates a high-performance pipeline for classifying toxic comments using a **binary classification** approach. The models were trained and evaluated using the **Jigsaw Toxic Comment Classification** dataset, specifically leveraging the domain-specific **Toxic-BERT** model as a primary architecture.
+
+---
+
+## **Project Overview**
+* **Objective**: To build an efficient **binary toxicity classifier** using state-of-the-art NLP models.
+* **Model Type**: Binary classification (Toxic vs. Non-Toxic).
+* **Dataset**: Jigsaw Toxic Comment Classification Challenge.
+* **Scope**: Includes data visualization, model benchmarking, and size reduction for deployment.
+
+---
+
+## **Technical Workflow**
+
+### **1. Data Preprocessing & EDA**
+* **Labeling**: Multi-label categories (toxic, severe_toxic, obscene, threat, insult, identity_hate) were condensed into a single **binary 'is_toxic' label**.
+* **Balancing**: The dataset was sampled to include 16,000 toxic and 16,000 non-toxic comments to ensure a balanced 32,000-sample training set.
+* **Cleaning**: Newline characters were removed to standardize the text input for transformer tokenizers.
+* **Visualization**: Word clouds were generated for both classes to identify the most frequent terms associated with toxic and non-toxic speech.
+
+### **2. Embedding Benchmarking**
+The project evaluated 15 different embedding sets across two categories:
+* **Light Models**: Includes DistilBERT, MiniLM, ALBERT, and ELECTRA-Small.
+* **Heavy Models**: Includes BERT, RoBERTa, DeBERTa, XLNet, and domain-specific models like **Toxic-BERT** and HateBERT.
+
+### **3. Model Performance Results**
+Models were evaluated using Logistic Regression (LR), Support Vector Machines (SVM), and Random Forest (RF).
+
+| Embedding | LR_AUC | LinearSVM_ACC | RBF_AUC | RF_ACC |
+| :--- | :--- | :--- | :--- | :--- |
+| **Toxic-BERT_transformer_emb** | 0.997022 | 0.979531 | 0.991532 | 0.979375 |
+| **HateBERT_transformer_emb** | 0.967701 | 0.901875 | 0.965530 | 0.852344 |
+| **DistilBERT_transformer_emb** | 0.967614 | 0.898906 | 0.967362 | 0.878125 |
+
+---
+
+## **Optimization Techniques**
+
+### **4. Dynamic Quantization**
+To optimize the teacher model (Toxic-BERT) for CPU inference, dynamic quantization was applied to convert weights from FP32 to INT8.
+* **Size Reduction**: The model size decreased from **438.01 MB** to **181.49 MB**.
+* **Accuracy Retention**: The quantized model maintained a high **Test AUC of 0.9966**, showing negligible performance loss despite the 58% reduction in size.
+
+### **5. Knowledge Distillation**
+A smaller student model (**DistilBERT**) was trained to mimic the behavior of the **Toxic-BERT** teacher.
+* **Loss Function**: A custom **Binary Knowledge Distillation** loss was used, combining Kullback-Leibler (KL) divergence for soft teacher probabilities and Cross-Entropy for hard labels.
+* **Student Performance**: Reached a **Validation AUC of 0.9866** after 5 training epochs.
+* **Final Footprint**: The student model is **267.86 MB**, significantly more portable than the original **438.03 MB** teacher model.
+
+---
+
+## **Requirements**
+* `torch`
+* `transformers`
+* `sentence-transformers`
+* `pandas`, `numpy`
+* `matplotlib`, `wordcloud`
+* `scikit-learn`