discord-moderation-minilm-l12

A 34 MB INT8 ONNX classifier for chat moderation. This is the accuracy variant of a pair:

	size	latency	macro F1	false positives
discord-moderation-minilm (L6)	23 MB	~2 ms	0.982	2.7%
this model (L12)	34 MB	~3.7 ms	0.987	2.1%

Pick L6 if you're on a Raspberry Pi or running at very high volume. Pick this one otherwise — it's 1.5x the size for a measurably lower false-positive rate, and 3.7 ms is still nothing.

Classes	benign · abusive · spam · flood
Base	sentence-transformers/all-MiniLM-L12-v2 (33M params)
Context	128 tokens

Why this exists

Most open toxicity classifiers have two problems for chat moderation:

1. No spam or flood class. They're toxicity-only, so a pipeline has to bolt on a separate heuristic. This model predicts all four classes natively.
2. They over-flag competitive banter. "I'm gonna destroy you in Rocket League" is not a threat. False positives punish innocent users, and they're what gets a moderation bot uninstalled.

Results

Held-out test set of 2,641 messages the model never trained on. Threshold 0.85.

metric	value
macro F1	0.987
false positives (innocent messages flagged)	24/1,167 = 2.1%
false negatives (abuse missed)	32/1,474 = 2.2%
out-of-distribution suite (39 hand-written cases: short messages, banter, threats)	39/39
latency	~3.7 ms/msg, INT8 ONNX on desktop CPU

Cross-domain, measured honestly (external datasets the model never saw):

domain	result
Reddit comments removed by real human moderators	~99% caught
real game chat (CONDA valid): explicit toxicity	74.3% caught
real game chat: normal messages	89.6% left alone
real game chat: action words (gg, glhf)	90.0% left alone

The game-chat rows are the honest weakness. This model is trained on Wikipedia-register text and transfers well to Reddit-like registers — not to gaming chat. If you deploy in a gaming community, fine-tune on domain data: in our experiments, ~9k rows of real game chat lifted implicit game-toxicity recall from 14% → 77%. Those rows aren't in the published weights for licensing reasons (this model's data diet is strictly CC0 + owned synthetic).

For scale: general-purpose LLM judges prompted for moderation — the usual alternative — measured far worse on the same yardsticks at 30–70x the size (phi3:mini, 2.2 GB: F1 82, ~27% false positives; llama-guard3:1b: missed 70% of interpersonal abuse).

Usage

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer, pipeline

REPO = "strictlyinsecure/discord-moderation-minilm-l12"
model = ORTModelForSequenceClassification.from_pretrained(REPO, subfolder="onnx", file_name="model.onnx")
tok = AutoTokenizer.from_pretrained(REPO)
clf = pipeline("text-classification", model=model, tokenizer=tok, top_k=None, device=-1)

clf("i'm gonna destroy you in rocket league tonight lmao")   # → benign
clf("FREE NITRO for everyone, claim here bit.ly/x9k2")       # → spam
clf("shut up you idiot nobody likes you")                    # → abusive

Requires pip install optimum[onnxruntime]. A PyTorch fp32 checkpoint (model.safetensors) is also included if you want to fine-tune further.

Thresholding

The model is bimodal — it outputs ~0.99 or ~0.00 and little in between, so the threshold is not a delicate choice. 0.85 is a good default; raise it toward 0.95 if a false positive triggers a severe action (ban), lower it toward 0.6 if a human reviews every flag.

Training

• 4-class head on all-MiniLM-L12-v2, max_length=128, class-weighted loss (inverse frequency — without it the model collapses to predicting abusive for everything).
• ~18.6k rows: stratified Jigsaw (CC0)
  • synthetic chat-register data (spam, flood, evasion, short messages, hard benign negatives).
• Length-matched sampling. An early version learned "short message ⇒ abusive" because Wikipedia's toxic comments are short and its clean ones are long — it scored 97% held-out while classifying "great job everyone" as abusive. Every class is now sampled to the same length distribution, and a hand-written out-of-distribution suite exists to catch exactly this.
• Label denoising. 2,466 Jigsaw rows whose human label was confidently contradicted by an LLM second-annotator across the benign/abusive boundary were dropped. This measurably lowered false positives without hurting recall at matched operating points.

toxicity and harassment are merged into one abusive class: two annotators (Jigsaw's humans and an LLM judge) disagreed about which bucket an abusive message belonged in 48% of the time. That boundary is noise, and most pipelines take the same action for both.

Limitations

• Spam/flood scores are optimistic — trained and evaluated on templated synthetic data. Read them as "learned these patterns", not "solved spam".
• Never trained on real chat logs. See the game-chat numbers above.
• English only.
• Evasion coverage is synthetic-only (leetspeak, homoglyphs, spacing) — a determined evader will get through.
• Evaluated on the author's own held-out split, not a public benchmark.

Intended use

A fast first-pass filter in a moderation pipeline — cheap enough to run on every message, with a human or a stronger model handling escalation. Not a substitute for human moderators, and not a sound basis for automated bans on its own.

License & attribution

Apache-2.0. See NOTICE for the attribution required on redistribution.

Trained on the Jigsaw Toxic Comment Classification dataset (CC0); underlying comment text originates from Wikipedia and is licensed CC-BY-SA-3.0. Base model is Apache-2.0.