| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: transformers |
| tags: |
| - incivility |
| metrics: |
| - f1 |
| widget: |
| - text: "Be careful around those DemocRats." |
| example_title: "Namecall" |
| - text: "Be careful around those Democrats." |
| example_title: "No Namecall" |
| --- |
| |
| # Model Card for roberta-base-namecalling |
|
|
| This is a [roBERTa-base](https://huggingface.co/roberta-base) model fine-tuned on ~12K social media posts annotated for the presence or absence of namecalling. |
|
|
| # How to Get Started with the Model |
|
|
| You can use this model directly with a pipeline for text classification: |
|
|
| ```python |
| >>> import transformers |
| >>> model_name = "civility-lab/roberta-base-namecalling" |
| >>> classifier = transformers.TextClassificationPipeline( |
| ... tokenizer=transformers.AutoTokenizer.from_pretrained(model_name), |
| ... model=transformers.AutoModelForSequenceClassification.from_pretrained(model_name)) |
| >>> classifier("Be careful around those Democrats.") |
| [{'label': 'not-namecalling', 'score': 0.9995089769363403}] |
| >>> classifier("Be careful around those DemocRats.") |
| [{'label': 'namecalling', 'score': 0.996940016746521}] |
| ``` |
|
|
| # Model Details |
|
|
| This is a 2023 update of the model built by [Ozler et al. (2020)](https://aclanthology.org/2020.alw-1.4/) incorporating data from [Rains et al. (2021)](https://doi.org/10.1093/hcr/hqab009) and using a more recent version of the transformers library. |
|
|
| - **Developed by:** |
| [Steven Bethard](https://bethard.github.io/), |
| [Kate Kenski](https://comm.arizona.edu/user/kate-kenski), |
| [Steve Rains](https://comm.arizona.edu/user/steve-rains), |
| [Yotam Shmargad](https://www.yotamshmargad.com/), |
| [Kevin Coe](https://faculty.utah.edu/u0915886-Kevin_Coe/) |
| - **Language:** en |
| - **License:** apache-2.0 |
| - **Parent Model:** roberta-base |
| - **Resources for more information:** |
| - [GitHub Repo](https://github.com/clulab/incivility) |
| - Kadir Bulut Ozler; Kate Kenski; Steve Rains; Yotam Shmargad; Kevin Coe; and Steven Bethard. [Fine-tuning for multi-domain and multi-label uncivil language detection](https://aclanthology.org/2020.alw-1.4/). In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 28–33, Online, November 2020. Association for Computational Linguistics |
| - Stephen A Rains; Yotam Shmargad; Kevin Coe; Kate Kenski; and Steven Bethard. [Assessing the Russian Troll Efforts to Sow Discord on Twitter during the 2016 U.S. Election](https://doi.org/10.1093/hcr/hqab009). Human Communication Research, 47(4): 477-486. 08 2021. |
| - Stephen A Rains; Jake Harwood; Yotam Shmargad; Kate Kenski; Kevin Coe; and Steven Bethard. [Engagement with partisan Russian troll tweets during the 2016 U.S. presidential election: a social identity perspective](https://doi.org/10.1093/joc/jqac037). Journal of Communication, 73(1): 38-48. 02 2023. |
|
|
| # Uses |
|
|
| The model is intended to be used for text classification, taking as input social media posts and predicting as output whether the post contains namecalling. |
|
|
| It is not intended to generate namecalling, and it should not be used as part of any incivility generation model. |
|
|
| # Training Details |
|
|
| The model was trained on data from four sources: comments on the Arizona Daily Star website from 2011, Russian troll Tweets from 2012-2018, Tucson politician Tweets from 2018, and US presidential primary Tweets from 2019. |
| Each dataset was annotated for the presence of namecalling following the approach of [Coe et al. (2014)](https://doi.org/10.1111/jcom.12104) and split into training, development, and test partitions. |
|
|
| The [roberta-base](https://huggingface.co/roberta-base) model was fine-tuned on the combined training partitions from all four datasets, with texts tokenized using the standard [roberta-base](https://huggingface.co/roberta-base) tokenizer. |
| |
| # Evaluation |
|
|
| The model was evaluated on the test partition of each of the datasets. It achieves the following F1 scores: |
|
|
| - 0.58 F1 on Arizona Daily Star comments |
| - 0.71 F1 on Russian troll Tweets |
| - 0.71 F1 on Tucson politician Tweets |
| - 0.81 F1 on US presidential primary Tweets |
|
|
| # Limitations and Biases |
|
|
| The human coders and their trainers were mostly [Western, educated, industrialized, rich and democratic (WEIRD)](https://www.nature.com/articles/466029a), which may have shaped how they evaluated incivility. |
| The trained models will reflect such biases. |
|
|
| # Environmental Impact |
|
|
| - **Hardware Type:** Tesla V100S-PCIE-32GB |
| - **Hours used:** 22 |
| - **HPC Provider:** <https://hpc.arizona.edu/> |
| - **Carbon Emitted:** 2.85 kg CO2 (estimated by [ML CO2 Impact](https://mlco2.github.io/impact#compute)) |
|
|
| # Citation |
|
|
| ```bibtex |
| @inproceedings{ozler-etal-2020-fine, |
| title = "Fine-tuning for multi-domain and multi-label uncivil language detection", |
| author = "Ozler, Kadir Bulut and |
| Kenski, Kate and |
| Rains, Steve and |
| Shmargad, Yotam and |
| Coe, Kevin and |
| Bethard, Steven", |
| booktitle = "Proceedings of the Fourth Workshop on Online Abuse and Harms", |
| month = nov, |
| year = "2020", |
| address = "Online", |
| publisher = "Association for Computational Linguistics", |
| url = "https://aclanthology.org/2020.alw-1.4", |
| doi = "10.18653/v1/2020.alw-1.4", |
| pages = "28--33", |
| } |
| ``` |
|
|
