metadata
language: en
license: mit
base_model: distilbert/distilbert-base-uncased
pipeline_tag: text-classification
tags:
- toxicity
- content-moderation
datasets:
- google/jigsaw_unintended_bias
DistilBERT toxicity classifier (soft labels + identity weights)
Fine-tuned distilbert-base-uncased that predicts whether an English comment is toxic. Built for the capstone project
Trust in Automated Moderation of Toxic Comments (ECE 4424 / CS 4824). Code: https://github.com/martinagg7/ML-ToxicModeration
Training
- Data: Jigsaw Unintended Bias in Toxicity Classification (
train.csv, 1.8 M news comments), 80/10/10 split grouped by article. - Soft labels: the model learns the share of annotators who marked a comment as toxic, not only 0/1.
- Identity weights: comments that mention an identity weigh more in the loss (2 if toxic, 3 if non-toxic).
- 2 epochs, learning rate 2e-5, batch size 16, max length 128 tokens. Epoch chosen on validation PR-AUC.
Results (held-out test set, 177,635 comments, threshold 0.5)
| Model | Precision | Recall | F1 | ROC AUC | PR-AUC |
|---|---|---|---|---|---|
| Word blocklist (LDNOOBW) | 0.27 | 0.10 | 0.15 | 0.54 | 0.10 |
| TF-IDF + logistic regression | 0.73 | 0.53 | 0.61 | 0.93 | 0.69 |
| This model | 0.80 | 0.58 | 0.67 | 0.96 | 0.78 |
Limitations
- Misses about 4 in 10 toxic comments at threshold 0.5.
- Still blocks more innocent comments when they mention an identity (for example 5.6% for gay/lesbian vs 1.3% overall).
- Fails on negations with insults ("You are not an idiot"), counter-speech and positive swearing.
- Evaluated only on English news comments from the same source as training.
Not intended for fully automatic moderation: uncertain predictions (probability 0.4–0.6) should go to human review.
Usage
from transformers import pipeline
clf = pipeline("text-classification", model="Martinagg/toxic-distilbert")
clf("You are an idiot")