foggughost0/Hybrid_RoBERTa_Detector_for_LLM_generated_Texts

Hybrid RoBERTa + Linguistic-Features Detector

Best checkpoint from the master's thesis on hybrid machine learning approaches for detection of LLM-generated English texts by Bohdan Zhvalevskyi. It combines a fine-tuned roberta-base CLS embedding with 25 normalized linguistic features through a feature-attention module and a learned gated fusion, followed by a 2-class classifier (0 = human, 1 = machine).

Model details

• Developed by: Bohdan Zhvalevskyi (master's thesis)
• Model type: Hybrid transformer + hand-crafted linguistic-feature classifier
• Task: Binary detection of machine-generated vs. human-written English text
• Language: English
• Base model: FacebookAI/roberta-base (bottom 6 layers + embeddings frozen)
• License: Apache-2.0
• Architecture: RoBERTa CLS embedding + 25 linguistic features → feature attention → linear projection → learned gated fusion → 2-class head (see model.py).
• The original training pipeline is found on github for reproducibility: link to github repo

Intended uses & limitations

Intended use. Research on LLM-generated text detection and adversarial robustness; scoring English documents/paragraphs as human- or machine-written.

Limitations & biases.

• English only. Trained on English (DETree, RAID subset); not validated on other languages or code.
• Domain/generator shift. Trained on a fixed set of generators and attack types (see below). Newer models or unseen attacks may degrade performance.
• Not for high-stakes decisions. No detector is reliable enough to be the sole basis for consequential judgements (e.g. academic misconduct). Treat outputs as a probabilistic signal, not proof.
• Requires the exact feature pipeline. Predictions are only valid when the 25 features are extracted with features.py and normalized with the shipped ling_scaler.pkl; a raw/zero feature vector produces meaningless scores.

Files

File	Purpose
hybrid_model_best.pt	PyTorch state_dict for HybridClassifier (~500 MB).
model.py	Self-contained architecture definition + load_model() helper.
features.py	Self-contained linguistic feature extraction (normalize → chunk → 25 features).
ling_scaler.pkl	Fitted training StandardScaler used to normalize the 25 features. Required for valid predictions.
example_usage.py	Runnable end-to-end scoring example (raw text → prediction).

Install

pip install torch transformers spacy scikit-learn
python -m spacy download en_core_web_lg

Quick start

Score raw text end-to-end (extraction + normalization + document-level scoring are handled for you):

python example_usage.py --text "Your transcript goes here."
python example_usage.py --file document.txt

Or from Python:

import torch
from transformers import RobertaTokenizer, GPT2LMHeadModel, GPT2TokenizerFast
import spacy, pickle
from model import load_model, CONFIG
from features import extract_raw_features, prepare_document
from example_usage import score_text

device = "cuda" if torch.cuda.is_available() else "cpu"
model = load_model("hybrid_model_best.pt", device=device)
tokenizer = RobertaTokenizer.from_pretrained(CONFIG["roberta_model"])
nlp = spacy.load("en_core_web_lg", disable=["ner"])
gpt2_tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")
gpt2_model = GPT2LMHeadModel.from_pretrained("gpt2").to(device).eval()
scaler = pickle.load(open("ling_scaler.pkl", "rb"))

p_llm, chunk_probs = score_text(
    "Your transcript goes here.",
    model, tokenizer, nlp, gpt2_model, gpt2_tokenizer, scaler, device=device,
)
print(p_llm, "->", "machine" if p_llm >= 0.5 else "human")

Linguistic features

The 25 features, in the exact order the model expects, are extracted by features.py and normalized with the fitted training scaler (ling_scaler.pkl):

msttr, avg_word_len, hapax_ratio, function_ratio, punct_density, char_entropy,
burstiness, repetition_ratio, avg_sent_len, sent_len_std, noun_ratio,
verb_ratio, adj_ratio, adv_ratio, pron_ratio, pos_diversity, avg_tree_depth,
max_tree_depth, sub_clause_ratio, dm_density, sent_len_cv, fp_ratio,
num_sentences, words_per_sent, perplexity

features.py reproduces the training/testing pipeline exactly: normalize_text → sliding_window_chunk (450-word windows, 350-word stride) → 24 spaCy features

• GPT-2 perplexity → StandardScaler.transform. Document-level scores are the mean of per-chunk P(machine).

Training data

Trained on the DETree dataset (RAID subset), balanced across six conditions to study adversarial robustness: no_attack (clean), synonym, polish, paraphrase, perplexity_attack, and paraphrase_by_llm. Labels are 0 = human, 1 = machine. Documents are lowercased/cleaned, split by document id (no document leakage across train/val/test), then chunked with a 450-word sliding window (350-word stride). See 01_data_preprocessing_v2.py.

Training procedure

• Base roberta-base with embeddings and the bottom 6 encoder layers frozen.
• 25 linguistic features (StandardScaler-normalized) passed through a feature-attention module and projected to 768-d, then fused with the RoBERTa CLS embedding via a learned sigmoid gate.
• Dropout 0.15; max sequence length 512; seed 42.

Evaluation results

Document-level metrics on the held-out test set (6,292 documents / 6,806 chunks), aggregation = mean chunk probability per document, thresholded at 0.5.

Model	Accuracy	Macro-F1	AUC
Hybrid (this model)	0.9507	0.9505	0.9772
RoBERTa-only baseline	0.9170	0.9158	0.9824
Feature-only MLP baseline	0.8462	0.8459	0.9213

Robustness (Macro-F1 by attack type). The hybrid's margin over RoBERTa is largest on the hardest attacks — Polish (0.896 vs 0.798) and Paraphrase (0.926 vs 0.861) — because the linguistic features restore specificity that RoBERTa loses on attacked human text. The improvement over RoBERTa is statistically significant (McNemar p < 1e-12; ΔMacro-F1 = +0.035, 95% CI [+0.028, +0.041]).

Notes

• The checkpoint is a plain state_dict; load it with the HybridClassifier in model.py (see load_model()).
• Long documents were chunked to 512 tokens at training time; the reported document-level score is the mean chunk probability per document.
• return_gate=True in forward() also returns the fusion gate values and the feature-attention weights for interpretability.