README.md

metadata

license: mit
language:
  - en
tags:
  - text
  - conversations
  - classification
pretty_name: Slop classifier dataset

Slop classifier dataset

A human-annotated dataset for studying and classifying AI-generated text that people perceive as “AI slop.”

The dataset is built from samples collected from existing public datasets and annotated through the Bench Labs SlopFinder interface.

Slop score

Each sample receives a score based on human votes:

-1 = definitely slop
 0 = undecided / neutral
+1 = not slop at all

The score represents human judgment, not an objective measure of quality, AI-generatedness, or factual correctness.

Multiple people may vote on the same sample. This allows agreement and disagreement between annotators to be preserved rather than forcing every sample into a single binary label.

Data

Each annotation contains information such as:

• content — the text being evaluated
• slop_score — the submitted human rating
• source_dataset — the dataset the sample originated from
• source_row_id — the original sample identifier
• content_hash — identifier used to help detect duplicate content
• created_at — time the annotation was collected

Source samples may contain conversations rather than standalone text.

Collection

Samples are selected from public datasets and presented randomly to contributors through SlopFinder.

Contributors are given a single continuous slider rather than a list of predefined categories. This is intended to make annotation quick and reduce the friction of contributing.

No account is required to submit a vote.

The dataset is continuously growing as more annotations are collected.

Intended use

This dataset is intended for:

• training AI-slop classification models
• studying human perception of AI-generated writing
• evaluating whether text classifiers generalize across different sources
• research into stylistic characteristics associated with perceived AI slop

It should not be treated as a definitive benchmark for writing quality or AI-generatedness.

Limitations

The labels represent subjective human judgments. Different contributors may have very different ideas of what constitutes “slop.”

The dataset may also contain biases introduced by:

• the source datasets used for sampling
• the population of contributors
• differences in individual definitions of “slop”
• repeated exposure to similar types of text

A score should therefore be interpreted as perceived sloppiness, rather than an objective property of the text.

Contribute

This project is an early preview, and contributions are welcome.

Try SlopFinder and help expand the dataset:

Bench Labs Slop Classifier

License

MIT