nakas/gpttrace-v8s-1b

GPTTrace text detector: Ettin-1B (v8s)

A binary classifier that tells human writing apart from machine-generated text, including text run through "AI humanizers" and text from 2026 models (GPT-6.x, Claude Opus 5.5, Gemini 3, Qwen3). It replaces nakas/gpttrace-v5-1b, which was #2 on RAID.

v8s keeps v5's RAID accuracy and is far better on almost everything RAID does not measure: paraphrase and humanizer rewrites, newer generators, and false positives on ordinary human web writing (v5 flagged most pre-2022 web copy as AI at its raw cut-off; v8s flags 2.3%).

• Base model: jhu-clsp/ettin-encoder-1b (MIT)
• Outputs: two logits, [human, machine]; the score is softmax[...,1].
• Files: full precision (model/), fp32 ONNX (onnx-fp32/), 4-bit ONNX for CPU serving (onnx/, MatMulNBits block 32: AUROC 0.9955 vs 0.9958 in PyTorch on 300 MAGE texts, 99% sign agreement), and normalize.py.
• Code: github.com/andrewnakas/gpttrace (eval/train/text/, eval/bench/humbench/)
• Try it: gpttrace.com/ai-text-detector

Results

Our own runs of public benchmarks (scripts in eval/bench/humbench/). Thresholds for the TPR columns are set on each benchmark's own human texts; benchmarks without human texts use a fixed reference set of human literature, Wikipedia and PMC texts. Only benchmarks whose data licence allows commercial use are listed; we also track several research-only sets internally.

Benchmark	Metric	v5-1b	v8s	Best published we know of
RAID, held-out training titles (all attacks)	TPR @ 5% FPR	99.93	99.94	— (official test: see below)
MAGE, out-of-distribution paraphrased	AUROC	0.810	0.919	0.87 (DivEye, arXiv 2509.18880)
TextSight AI-text benchmark 2026	AUROC	0.863	0.999	0.867 (dataset card)
MELD-eval (GPT-5.4-mini, Gemini 3 Flash, Claude Haiku, Qwen 3.6; attacks)	TPR @ 1% FPR	99.90	100.0	99.9 (MELD paper)
UChicago/Booth detector benchmark (4 LLMs + StealthGPT)	AUROC	0.860	0.914	—
ARB (LLM rewrites of human and AI text)	AUROC	0.745	0.931	—
StealthRL (RL-trained paraphrase attack on MAGE)	TPR @ 5% FPR	0.902	0.936	—
OpAI-Bench (progressive human→AI co-editing, document level)	AUROC	0.608	0.765	—
AI Humanizer Benchmark + HumanizerBench (commercial humanizers)	TPR @ 5% FPR	0.956	0.791	—
Our humanizer holdout (Claude Opus 5.5 / GPT-6.x rewrite chains)	TPR @ 5% FPR	0.870	0.986	—
Our open-model humanizer holdout	TPR @ 5% FPR	0.868	0.980	—
Pre-2022 human web text (2,000 held-out FineWeb pages)	flagged as AI at logit 0	~60%	2.3%	—

Weak spots, stated plainly: commercial humanizer output is caught less often than by v5 at a strict threshold (0.79 vs 0.96 above); formal or academic human prose by non-native writers is flagged more often at the raw logit-0 cut-off, so use a calibrated threshold (below), not 0.

Calibrated threshold. Pooling the human texts of the benchmarks above, a logit of 9.3 flags 1% of human texts (10.7 for 0.5%). At 9.3, v8s still catches 93% of attacked RAID AI text and 0% of the reference, TextSight, web, UChicago and MELD human texts are flagged.

RAID official test set: predictions submitted to the leaderboard (liamdugan/raid); the score will be added here once the leaderboard CI reports it.

Training

• Start: a weight average of two earlier checkpoints, 0.7·v5 + 0.3·v6 (both Ettin-1B; v6 was fine-tuned from v5).
• Two fine-tunes from that start, one epoch each, lr 1e-5, cosine, batch 64, bf16 on one A100: a 0.44M-text run, and a 1.1M-text run that adds 26k pre-2022 web pages as human text. v8s is the 50/50 weight average of the two, which kept the larger run's web false-positive fix and most of the smaller run's humanizer recall.
• Data: RAID train (MIT; held-out titles never used) with hard examples mined by v5; MAGE train (Apache-2.0); Wikipedia, PMC open-access (commercial), arXiv abstracts, Gutenberg; pre-2022 FineWeb pages (ODC-BY); the public part of nakas/gpttrace-text-train (human/AI pairs, humanizer rewrites by open models, Claude and GPT-6.x); synthetic "mirror" AI texts written to match human texts' genre; and emulations of commercial humanizer operations. Text overlapping our evaluation sets (8-gram match) was dropped.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from normalize import normalize_text            # included in this repo

repo = "nakas/gpttrace-v8s-1b"
tok = AutoTokenizer.from_pretrained(repo, subfolder="model")
model = AutoModelForSequenceClassification.from_pretrained(repo, subfolder="model").eval()
enc = tok(normalize_text(text), truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits[0]
    logit = (logits[1] - logits[0]).item()      # compare with the calibrated threshold (9.3 ≈ 1% FPR)

Licence and limits

• Licence: CC BY-SA 4.0. Part of the human training text is share-alike (Wikipedia), so the weights are released under the same terms. The base model is MIT.
• A detector score is evidence, not proof. Short texts (under ~150 words) are unreliable, and every detector can be evaded by determined rewriting. Never use a score alone to accuse someone.