vkusterl/toxic-steam-user-distilbert

Toxic Steam user classifier

This model reads the English reviews written by a Steam user and estimates whether that user belongs to the group associated with toxic review behavior. It is the end to end transformer of the paper Toxic Steam: A Large-Scale Characterization and Prediction of Toxic Users in a Gaming Platform, accepted at AIIDE 2026, retrained with the exact recipe of the paper on the same sample of 60,000 users and then evaluated on 60,000 other users that it never saw, at the real platform prevalence of about 4%.

The scores are meant to rank users for human review at platform scale. They are not a verdict on anyone's character and should never trigger an automatic sanction, for the reasons explained in the next section.

Content warning. The training data contains offensive and hateful language.

What the label measures

The study collected 36.8 million English reviews from 14.1 million Steam users through the public Steam Web API, between September 2024 and April 2025. Each review was scored by two automatic detectors, Perspective API and Detoxify, with thresholds calibrated on 400 reviews annotated by three people (Fleiss kappa of 0.85 for Detoxify and 0.75 for Perspective). A user is labeled toxic_user when at least one of their reviews is flagged.

Two consequences follow. The model learns to anticipate the decisions of those detectors, not human judgment, and it reads the same reviews that define the label, so most of its accuracy comes from finding the flagged text itself. In the paper, a leave toxic out control that removes the flagged reviews before representing each user still beats profile metadata (AUC-PR of 0.166 against 0.063 with XGBoost), which shows that a weaker behavioral trace exists beyond the flagged text, but far below the full score. The label also grows with activity, since users who write more reviews have more chances of being flagged.

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="vkusterl/toxic-steam-user-distilbert", top_k=None)

def user_text(reviews):
    # same input format used in training
    return " [SEP] ".join(r[:500] for r in reviews)[:2048]

reviews = [
    "Great campaign with friends, terrible matchmaking.",
    "Worth it on sale. The developers actually listen to feedback.",
]
scores = {d["label"]: d["score"] for d in clf([user_text(reviews)], truncation=True, max_length=256)[0]}
threshold = clf.model.config.decision_threshold
print(round(scores["toxic_user"], 3), scores["toxic_user"] >= threshold)

Each review is cut at 500 characters, the reviews are joined with [SEP] in order of publication, and the whole text is limited to 2,048 characters and 256 tokens. The threshold stored in the configuration (decision_threshold, 0.927) was chosen on a validation split to maximize macro F1. Whoever uses the scores should move it according to how many cases their reviewers can actually inspect.

Results

All numbers below keep the real prevalence. A random ranking has an AUC-PR equal to the prevalence itself.

Evaluation of this checkpoint on 60,000 users from the same 10% slice of the corpus who were not part of the training sample (2,383 toxic users, prevalence of 3.97%).

Metric	Value	95% bootstrap interval
AUC-PR	0.648	0.630 to 0.668
ROC-AUC	0.968	0.966 to 0.971
F1, toxic class	0.604	0.589 to 0.619
Precision, toxic class	0.526
Recall, toxic class	0.709
Macro F1	0.792

For reference, the paper reports the same recipe with ten fold cross validation on the 60,000 user sample, next to the other model families of the benchmark.

Model	AUC-PR	ROC-AUC	F1, toxic class
Profile metadata, LightGBM	0.070	0.637	0.101
Content embeddings, XGBoost	0.476	0.932	0.463
DistilBERT end to end, this recipe	0.643	0.968	0.605
Claude Opus as a classifier	0.408	0.921	0.389
GPT-4o as a classifier	0.313	0.688	0.405
Llama 3.1 8B as a classifier	0.308	0.818	0.344
Qwen 2.5 7B as a classifier	0.268	0.888	0.305

The classical models were evaluated with five folds on all 620,891 users of the slice, and the LLMs on only 1,000 users with 40 positives, so the LLM numbers carry much wider intervals. In the paper, the LLMs were more useful as generators of auditable explanations for human reviewers than as classifiers.

Results of this checkpoint by number of English reviews written by the user.

English reviews by the user	Users	Toxic users	Prevalence	AUC-PR	F1, toxic class
1	33,947	510	1.5%	0.685	0.644
2 to 4	19,411	905	4.7%	0.680	0.616
5 or more	6,642	968	14.6%	0.602	0.567

Users with a single review are the easiest group, because their label is decided by the very text the model reads. Prolific users are harder even though they are toxic far more often. Their input is cut at 256 tokens, so the reviews that define the label are often left out, and recall drops accordingly.

Training

Setting	Value
Base model	distilbert/distilbert-base-uncased
Data	60,000 users sampled at real prevalence from a 10% hash slice of 620,891 users with public profiles, 90% for training and 10% to choose the threshold
Balancing	Negatives undersampled to 1:1, 4,320 training examples
Optimization	AdamW, learning rate 2e-5, batch of 16, 2 epochs (540 steps)
Input	Up to 256 tokens per user
Hardware	NVIDIA GTX 1070, 7 minutes

The only change from the paper code is dynamic padding inside each batch, which does not change the computation and makes training faster. The scripts that rebuild this checkpoint and its evaluation are in the scripts folder (preparar.py and treinar_final.py). They need the original corpus, which is not distributed.

Limitations and responsible use

The labels come from automatic detectors and inherit their errors. Performative profanity and game vocabulary such as "kill" are known sources of false positives that calibration reduces but does not remove. The corpus is English only, comes from a single collection window and covers the 43.7% of users whose reviews could be matched to a public profile, so behavior in other languages, in chat or in discussion forums is outside its scope. The model classifies users within that window and does not forecast future behavior.

Any use for moderation should keep a person in the loop, report the label definition, the circularity and the uncertainty together with the scores, and treat a positive prediction as a reason to look closer, never as evidence of wrongdoing.

Data and code

The raw reviews and user data are not distributed. They were collected from the public Steam Web API, are subject to the Steam terms of service, and the people who wrote them did not consent to toxicity profiling. The full pipeline, the result tables and the documentation are in the GitHub repository.

Citation

@inproceedings{gibrim2026toxicsteam,
  title     = {Toxic Steam: A Large-Scale Characterization and Prediction of Toxic Users in a Gaming Platform},
  author    = {Gibrim, Paula T. M. and Lodi, Vinicius K. and Andrade, Gabriel S. and Ribeiro, Marcus V. G. and Gruppi, Maur{\'i}cio and Barbosa, Daniel M. and Melo, Philipe F. and Reis, Julio C. S.},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE)},
  year      = {2026}
}

Paula Gibrim and Vinicius Kuster Lodi share the first authorship. This checkpoint was trained and published by Vinicius Kuster Lodi, Universidade Federal de Viçosa.

Resumo em português

Modelo DistilBERT que lê as avaliações em inglês de um usuário da Steam e estima se ele pertence ao grupo associado a avaliações tóxicas. É o modelo final do artigo Toxic Steam, aceito no AIIDE 2026, treinado com a mesma receita do artigo e avaliado em 60 mil usuários inéditos na prevalência real de cerca de 4%, com AUC-PR de 0.648 contra 0,07 dos metadados de perfil. O rótulo vem de detectores automáticos e é em grande parte circular em relação ao texto lido pelo modelo, por isso o uso indicado é a triagem para revisão humana, nunca uma decisão automática.