data/README.md

Data module

Owns all data loading, cleaning, and the unified schema. Everything downstream imports from here.

Unified schema

from data import Argument, Debate

debate = Debate(id="abc", title="CMV: cats > dogs", source="cmv")
arg   = Argument(id="1", text="Cats are independent.", arg_type="premise")

arg_type must be one of: claim, counter_claim, premise, unknown.

Sources

CMV (offline)

Download the Tan et al. 2016 dataset:

mkdir -p data/raw/cmv
curl -O https://chenhaot.com/data/cmv/cmv.tar.bz2
tar -xjf cmv.tar.bz2 -C data/raw/cmv/

Then:

from data import load_cmv, clean_debates
debates = clean_debates(load_cmv("train"))

IBM Debater (auto-download)

from data import load_ibm
debates = load_ibm("train")  # downloads via HuggingFace datasets on first run

IBM Claim Stance (auto-download)

~2,400 claim–topic pairs labeled PRO or CON against debate motions. Directly provides claim vs. counter-claim ground truth.

from data import load_claim_stance
debates = load_claim_stance("train")

Preprocessing

clean_debates() applies Reddit-specific noise removal (quoted text, URLs, edit notes, user mentions) and drops arguments shorter than 5 tokens. Call it on any list of Debate objects before passing them downstream.