GPT-2 Fixed-Position AI-Text Detector

This is a fully fine-tuned GPT-2 classifier for distinguishing human-written and AI-generated text. It uses a fixed-position readout token at the final position of a 1,024-token context. The model was trained on rasbt/human-vs-ai-50k. Human-written text has label 0 and AI-generated text has label 1.

Temperature scaling is applied during inference. The recorded best validation accuracy was 96.26%.

Download and use

hf download rasbt/ai-text-detector-gpt2-fixed \
  --local-dir models/ai-text-detector-gpt2-fixed

import json
from pathlib import Path

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer


model_dir = Path("models/ai-text-detector-gpt2-fixed")
metadata = json.loads(
    (model_dir / "detector-config.json").read_text(encoding="utf-8")
)
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForSequenceClassification.from_pretrained(model_dir)
model.eval()

text = "Paste the text to classify here."
text_ids = tokenizer(
    text,
    add_special_tokens=False,
    truncation=True,
    max_length=metadata["max_text_length"],
)["input_ids"]

if metadata["readout_position"] == "fixed":
    padding_length = metadata["context_length"] - len(text_ids) - 1
    input_ids = (
        text_ids
        + [tokenizer.pad_token_id] * padding_length
        + [tokenizer.eos_token_id]
    )
    attention_mask = [1] * len(text_ids) + [0] * padding_length + [1]
else:
    input_ids = text_ids + [tokenizer.eos_token_id]
    attention_mask = [1] * len(input_ids)

inputs = {
    "input_ids": torch.tensor([input_ids]),
    "attention_mask": torch.tensor([attention_mask]),
}
with torch.inference_mode():
    logits = model(**inputs).logits / metadata["temperature"]
    probabilities = logits.float().softmax(dim=-1)

ai_index = metadata["label_mapping"]["ai"]
ai_probability = probabilities[0, ai_index].item()
print({"score": round(100 * ai_probability, 4)})

Test-set confusion matrix

detector-config.json contains the readout, calibration, and training metadata. The recommended inference implementation is provided in the rasbt/ai-detector repository because classification requires selecting the configured readout position.

Related models

Limitations

Performance may change for text from generators, domains, languages, and editing workflows not represented in the training set. Short or partly AI-assisted text may also be harder to classify. The score should not be treated as definitive evidence that a person did or did not write a text.