discord-moderation-minilm

A 23 MB INT8 ONNX text classifier for chat moderation, built for edge deployment — it runs on a Raspberry Pi, not just a GPU box.

Classes	benign · abusive · spam · flood
Size	23 MB (INT8 ONNX) · 91 MB (fp32 safetensors)
Latency	~14 ms/message, desktop CPU, one message at a time
Base	sentence-transformers/all-MiniLM-L6-v2 (22M params)
Context	128 tokens

Why this exists

Most open toxicity classifiers have two problems for chat moderation:

1. No spam or flood class. They're toxicity-only, so a moderation pipeline has to bolt on a separate heuristic. This model predicts all four classes natively.
2. They over-flag competitive banter. "I'm gonna destroy you in Rocket League" is not a threat, but a model trained on Wikipedia edit-war comments often thinks it is. False positives punish innocent users, and they're the reason moderation bots get uninstalled.

Results

Held-out set of 1,200 messages the model never trained on (899 real Jigsaw rows, 301 synthetic). Classifier thresholded at 0.85. "False positives" = innocent messages wrongly flagged, out of 399 benign.

model	size	abuse caught	false positives	ms/msg
this model	23 MB	97.4%	26 / 399	14
gemma2:2b (LLM judge)	1.6 GB	90.5%	136 / 399	226
qwen2.5:1.5b (LLM judge)	1.0 GB	85.1%	117 / 399	72
phi3:mini (LLM judge)	2.2 GB	79.4%	109 / 399	194
llama-guard3:1b	1.6 GB	39.1%	80 / 399	47

Held-out macro F1: 0.975.

The comparison LLMs are general-purpose judges prompted for moderation, not fine-tuned — which is how most people actually deploy them. They are far larger and far slower, and they block 4–5x more innocent messages.

Identity-based hate: 68/68 caught (100%) on the held-out identity_hate rows — slurs are the easiest class for this model, not the hardest. Training includes 356 rows of real, human-written identity hate from Jigsaw.

Usage

ONNX (recommended — this is the 23 MB path)

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

REPO = "strictlyinsecure/discord-moderation-minilm"
model = ORTModelForSequenceClassification.from_pretrained(REPO, subfolder="onnx", file_name="model.onnx")
tok = AutoTokenizer.from_pretrained(REPO)
clf = pipeline("text-classification", model=model, tokenizer=tok, top_k=None, device=-1)

clf("i'm gonna destroy you in rocket league tonight lmao")
# → benign  (0.992)
clf("FREE NITRO for everyone, claim here bit.ly/x9k2")
# → spam    (0.993)
clf("shut up you idiot nobody likes you")
# → abusive (0.991)

Requires pip install optimum[onnxruntime].

PyTorch (fp32)

from transformers import pipeline

clf = pipeline("text-classification", model="strictlyinsecure/discord-moderation-minilm", top_k=None)
clf("FREE NITRO click here bit.ly/x9k2")
# [{'label': 'spam', 'score': 0.99}, ...]

Thresholding

The model is bimodal — it outputs ~0.99 or ~0.003 and little in between. On the held-out set, the trade-off curve is nearly flat between 0.3 and 0.9, so the threshold is not a delicate choice:

threshold	abuse caught	false positives
0.30	98.8%	38 / 399
0.85	97.4%	26 / 399
0.95	95.6%	20 / 399

Training

• Base: all-MiniLM-L6-v2, 4-class sequence classification head.
• Data: 8,000 rows — 6,000 stratified from Jigsaw Toxic Comment Classification (CC0), length-filtered to chat-sized messages, plus 2,000 synthetic chat-register rows (evasion, sarcasm, spam, flood, and hard benign negatives).
• Class-weighted loss (inverse frequency). Without it the model collapses to predicting abusive for everything — an early run scored 100% recall on abuse with 8% recall on benign, i.e. it flagged 92% of innocent messages.
• 70/15/15 split, early stopping on validation macro F1, INT8 dynamic quantization for export.
• INT8 cost 0.008 macro F1 versus fp32 — effectively free.

toxicity and harassment are merged into one abusive class. On the labeled corpus, two annotators (Jigsaw's humans and an LLM judge) disagreed about which of those two buckets an abusive message belonged in 48% of the time. That boundary was training noise, not signal, and most moderation pipelines take the same action for both.

Limitations — please read before deploying

• The spam and flood scores are optimistic. Those classes were trained on templated synthetic data, and the evaluation set draws from the same template families. They score ~100% here. Real-world spam is far more varied. Read this as "learned these patterns", not "solved spam".
• Never trained on real chat logs. The training data is Wikipedia comments plus synthetic chat. The register gap is real and measurable — see ToxBuster, which found +57 points of recall from training on actual game chat instead of Wikipedia. Expect degradation on live traffic.
• English only. Slurs and abuse in other languages will likely be missed.
• Evasion is only lightly covered. Leetspeak, unicode homoglyphs and spaced-out letters appear in training, but only from a synthetic generator — a determined evader will get through.
• Evaluated on the author's own held-out split, not a public benchmark. Numbers are not directly comparable to models evaluated elsewhere.

Intended use

A fast first-pass filter in a moderation pipeline — cheap enough to run on every message, with a human or a stronger model handling escalation. It is not a substitute for human moderators, and it should not be the sole basis for automated bans.

License & attribution

Apache-2.0. See NOTICE for the attribution required on redistribution.

Trained on the Jigsaw Toxic Comment Classification dataset (CC0). The underlying comment text originates from Wikipedia and is licensed CC-BY-SA-3.0.

Base model sentence-transformers/all-MiniLM-L6-v2 is Apache-2.0.