pandrei7/autextification-upb-mtl

UPB's Multi-task Learning model for AuTexTification

This is a model for classifying text as human- or LLM-generated.

This model was trained for one of University Politehnica of Bucharest's (UPB) submissions to the AuTexTification shared task.

This model was trained using multi-task learning to predict whether a text document was written by a human or a large language model, and whether it was written in English or Spanish.

The model outputs a score/probability for each task, but it also makes a binary prediction for detecting synthetic text, based on a threshold.

Training data

The model was trained on approximately 33,845 English documents and 32,062 Spanish documents, covering five different domains, such as legal or social media. The dataset is available on Zenodo (more instructions here).

Evaluation results

These results were computed as part of the AuTexTification shared task:

Language	Macro F1	Confidence Interval
English	65.53	(64.92, 66.23)
Spanish	65.01	(64.58, 65.64)

Using the model

You can load the model and its tokenizer using AutoModel and AutoTokenizer.

This is an example of using the model for inference:

import torch
import transformers

checkpoint = "pandrei7/autextification-upb-mtl"
tokenizer = transformers.AutoTokenizer.from_pretrained(checkpoint)
model = transformers.AutoModel.from_pretrained(checkpoint, trust_remote_code=True)

texts = [
    "You're absoutely right! Let's delve into it.",
    "Tengo monos en la cara.",
]
inputs = tokenizer(
    texts, padding=True, truncation=True, max_length=512, return_tensors="pt"
)

model.eval()
with torch.no_grad():
    preds = model(inputs)

for i, text in enumerate(texts):
    print(f"Text: '{text}'")
    print(f"Bot?          {preds['is_bot'][i].item()}")
    print(f"Bot score     {preds['bot_prob'][i].item()}")
    print(f"English score {preds['english_prob'][i].item()}")
    print()

Text: 'You're absoutely right! Let's delve into it.'
Bot?          True
Bot score     0.997463583946228
English score 0.9997979998588562

Text: 'Tengo monos en la cara.'
Bot?          False
Bot score     0.7036079168319702
English score 0.0002293310681125149