simonlesaumon/open-pangram-modernbert-hc3-detector

Open Pangram ModernBERT HC3 AI Text Detector

This repository contains the finished AI-vs-human text detector trained in the workspace. It is a PEFT LoRA adapter on top of answerdotai/ModernBERT-large, trained on a balanced HC3 split.

The uploaded files include:

• final PEFT adapter and tokenizer at the repository root;
• all training checkpoints from this run (checkpoint-50, checkpoint-100, checkpoint-150, checkpoint-200, checkpoint-225);
• optimizer/scheduler/RNG states inside checkpoints, so the run can be resumed;
• inference helper code under src/;
• training arguments and model card metadata.

Architecture

The model is a transformer sequence classifier:

• Base encoder: answerdotai/ModernBERT-large.
• Classification head: binary labels, human=0, ai=1.
• Fine-tuning method: PEFT LoRA / QLoRA.
• LoRA target modules inferred from ModernBERT: Wqkv, Wo, Wi, dense.
• LoRA rank: r=16.
• LoRA alpha: 32.
• LoRA dropout: 0.05.
• Quantization during training: 4-bit bitsandbytes QLoRA.
• Training precision: fp16 compute path, selected after bf16 produced non-finite logits in the local RTX 3090 environment.

Inference runs over token windows. The trained context length is 64 tokens. Longer texts are split into overlapping windows and the final AI probability is the mean window probability.

Training Setup

Dataset:

• Source: Hello-SimpleAI/HC3
• Balanced records: 40,000
• Train: 38,000
• Validation: 2,000
• Labels: human, ai

Run:

• Max length: 64
• Epochs: 3
• Per-device batch size: 128
• Gradient accumulation: 4
• Effective batch size: 512
• Learning rate: 2e-5
• Eval steps: 50
• Save steps: 50
• GPU used: RTX 3090 24 GB
• Train runtime: about 14 minutes 48 seconds

Important implementation note: a previous 512-token QLoRA attempt in this local environment produced NaN losses without Flash Attention. This final uploaded model intentionally uses 64-token windows because that configuration trained cleanly and evaluated well.

Validation Metrics

Validation set: 2,000 HC3 examples.

Metric	Value
Accuracy	0.9695
Precision	0.9440
Recall	0.9969
F1	0.9697
AUROC	0.9975
Average precision	0.9962

Checkpoint eval loss:

Checkpoint	Epoch	Eval loss
checkpoint-50	0.67	0.3219
checkpoint-100	1.34	0.0853
checkpoint-150	2.00	0.0537
checkpoint-200	2.67	0.0454
checkpoint-225	3.00	best model loaded from step 200

Best checkpoint: step 200 with eval_loss=0.045422744.

Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

repo_id = "simonlesaumon/open-pangram-modernbert-hc3-detector"
base_id = "answerdotai/ModernBERT-large"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
base_model = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    num_labels=2,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
)
model = PeftModel.from_pretrained(base_model, repo_id)
model.eval()
if torch.cuda.is_available():
    model.cuda()

text = "As an AI language model, I can provide a concise explanation."
encoded = tokenizer(
    [text],
    truncation=True,
    max_length=64,
    padding="max_length",
    return_tensors="pt",
)
if torch.cuda.is_available():
    encoded = encoded.to("cuda")

with torch.no_grad():
    probability_ai = torch.softmax(model(**encoded).logits, dim=-1)[0, 1].item()

print(probability_ai)

For long texts, use overlapping windows. See src/inference.py.

Local Gradio

pip install torch transformers peft gradio numpy
python src/app.py

Limitations

• The detector was trained and validated on HC3, mostly English question-answer style text.
• It should not be treated as proof that a specific person used AI.
• Thresholds should be calibrated for each deployment domain.
• The context length is intentionally short (64) because that was the stable training path in this environment; long documents are scored by window aggregation.
• Real-world AI-text detection is adversarial and can drift as generators change.