modernbert-ai-text-detector

Recommended general-purpose checkpoint: best cross-dataset transfer.

Fine-tuned ModernBERT-base classifier for detecting machine-generated English text (label 1 = machine). Built in the open research project robust-ai-text-detection, where every number regenerates from committed artifacts and the full research log, literature review, and negative results live.

Note. Strong cross-dataset transfer (M4GT), but weak on a 2025-generator probe in RAID's continuation format (TPR@1% 0.13) — no single checkpoint is robust on every axis; see the repository's research log and consider the companion raid-mix variant or a zero-shot ensemble for coverage.

Metrics

Thresholds must be calibrated on human text from your own distribution; the scores below use per-dataset calibration. AUROC alone is misleading for this task — use the low-FPR columns.

eval	AUROC	TPR@5%FPR	TPR@1%FPR
MAGE test (near in-distribution)	0.980	0.930	0.845
HC3 (cross-dataset)	0.994	0.970	0.941
RAID eval grid, incl. attacks (OOD)	0.875	0.512	0.310
M4GT (cross-dataset)	0.920	0.783	0.674
Frontier Qwen3-2025 probe (cross-generator)	0.881	0.297	0.126

Training

MAGE train (318K rows after artifact filtering), 1 epoch, lr 3e-5, 512 tokens.

Intended use and limitations

Research use. Output is probabilistic evidence, never proof of authorship: at 1% FPR, an institution processing 75,000 documents a year would wrongly flag ~750. Do not use as the basis of disciplinary action. Untested on non-native-writer false-positive rates; English only; document-level only; defeated by adaptive paraphrase attacks. Pair with Unicode/NFKC input normalization (see the repository) — it neutralizes homoglyph and zero-width attacks for free.

Citation

Please cite the datasets and methods this builds on (MAGE, RAID, ModernBERT — full BibTeX in the repository).