Anish-25/fact-and-logic

Fact and Logic: fallacy classifier

A DistilRoBERTa model fine-tuned to name the logical fallacy in a short piece of argument text. It covers 13 types, including ad hominem, false causality, false dilemma and faulty generalization. It's one half of Fact and Logic, a project that also checks claims against scientific abstracts.

• Code: https://github.com/Anishkumar25/fact-and-logic
• Demo: https://huggingface.co/spaces/Anish-25/fact-and-logic

Training

Fine-tuned from distilroberta-base on the tasksource/logical-fallacy dataset, using its own train, validation and test splits. Settings: 4 epochs, learning rate 3e-5, batch size 16, inputs cut at 256 tokens, seed 42. The checkpoint with the best validation macro-F1 was kept.

Results

On the 511-example test split: accuracy 0.44, macro-F1 0.40. Random guessing would score about 0.08, and always predicting the most common class about 0.17.

It does best on ad hominem (F1 0.71), false dilemma (0.57), ad populum (0.55) and faulty generalization (0.51). It does worst on equivocation (no correct predictions, but only 9 test examples) and fallacy of relevance (it finds 13% of them). Several of the broader categories overlap with each other, which makes them hard to separate.

Limits

• There is no "no fallacy" class, so the model will name a fallacy even for a plain factual sentence. In the demo app, guesses below a confidence cutoff are hidden.
• The training set is small and fallacy labels are partly a matter of judgment. Treat the output as a hint, not a ruling.
• English only, and best on short passages.