|
|
--- |
|
|
language: |
|
|
- en |
|
|
- es |
|
|
pipeline_tag: text-classification |
|
|
--- |
|
|
|
|
|
# UPB's Multi-task Learning model for AuTexTification |
|
|
|
|
|
This is a model for classifying text as human- or LLM-generated. |
|
|
|
|
|
This model was trained for one of University Politehnica of Bucharest's (UPB) |
|
|
submissions to the [AuTexTification shared |
|
|
task](https://sites.google.com/view/autextification/home). |
|
|
|
|
|
This model was trained using multi-task learning to predict whether a text |
|
|
document was written by a human or a large language model, and whether it was |
|
|
written in English or Spanish. |
|
|
|
|
|
The model outputs a score/probability for each task, but it also makes a binary |
|
|
prediction for detecting synthetic text, based on a threshold. |
|
|
|
|
|
## Training data |
|
|
|
|
|
The model was trained on approximately 33,845 English documents and 32,062 |
|
|
Spanish documents, covering five different domains, such as legal or social |
|
|
media. The dataset is available on Zenodo (more instructions |
|
|
[here](https://sites.google.com/view/autextification/data)). |
|
|
|
|
|
## Evaluation results |
|
|
|
|
|
These results were computed as part of the [AuTexTification shared |
|
|
task](https://sites.google.com/view/autextification/results): |
|
|
|
|
|
| Language | Macro F1 | Confidence Interval| |
|
|
|:---------|:--------:|:------------------:| |
|
|
| English | 65.53 | (64.92, 66.23) | |
|
|
| Spanish | 65.01 | (64.58, 65.64) | |
|
|
|
|
|
## Using the model |
|
|
|
|
|
You can load the model and its tokenizer using `AutoModel` and `AutoTokenizer`. |
|
|
|
|
|
This is an example of using the model for inference: |
|
|
|
|
|
```python |
|
|
import torch |
|
|
import transformers |
|
|
|
|
|
checkpoint = "pandrei7/autextification-upb-mtl" |
|
|
tokenizer = transformers.AutoTokenizer.from_pretrained(checkpoint) |
|
|
model = transformers.AutoModel.from_pretrained(checkpoint, trust_remote_code=True) |
|
|
|
|
|
texts = [ |
|
|
"You're absoutely right! Let's delve into it.", |
|
|
"Tengo monos en la cara.", |
|
|
] |
|
|
inputs = tokenizer( |
|
|
texts, padding=True, truncation=True, max_length=512, return_tensors="pt" |
|
|
) |
|
|
|
|
|
model.eval() |
|
|
with torch.no_grad(): |
|
|
preds = model(inputs) |
|
|
|
|
|
for i, text in enumerate(texts): |
|
|
print(f"Text: '{text}'") |
|
|
print(f"Bot? {preds['is_bot'][i].item()}") |
|
|
print(f"Bot score {preds['bot_prob'][i].item()}") |
|
|
print(f"English score {preds['english_prob'][i].item()}") |
|
|
print() |
|
|
``` |
|
|
|
|
|
```text |
|
|
Text: 'You're absoutely right! Let's delve into it.' |
|
|
Bot? True |
|
|
Bot score 0.997463583946228 |
|
|
English score 0.9997979998588562 |
|
|
|
|
|
Text: 'Tengo monos en la cara.' |
|
|
Bot? False |
|
|
Bot score 0.7036079168319702 |
|
|
English score 0.0002293310681125149 |
|
|
``` |
