| | --- |
| | license: cc-by-nc-4.0 |
| | language: |
| | - en |
| | metrics: |
| | - f1 |
| | pipeline_tag: text-classification |
| | tags: |
| | - transformers |
| | - argument-mining |
| | - opinion-mining |
| | - information-extraction |
| | - inference-extraction |
| | - Twitter |
| | widget: |
| | - text: 'Men shouldn’t be making laws about women’s bodies #abortion #Texas' |
| | example_title: Statement |
| | - text: >- |
| | ’Bitter truth’: EU chief pours cold water on idea of Brits keeping EU |
| | citizenship after #Brexit HTTPURL via @USER |
| | example_title: Notification |
| | - text: >- |
| | Opinion: As the draconian (and then some) abortion law takes effect in |
| | #Texas, this is not an idle question for millions of Americans. A slippery |
| | slope towards more like-minded Republican state legislatures to try to |
| | follow suit. #abortion #F24 HTTPURL |
| | example_title: Reason |
| | - text: '@USER Blah blah blah blah blah blah' |
| | example_title: None |
| | - text: republican men and karens make me sick |
| | example_title: Unlabeled 1 |
| | - text: No empire lives forever! Historical fact! GodWins! 🙏💪🇺🇲 |
| | example_title: Unlabeled 2 |
| | - text: >- |
| | Further author information regarding registration and visa support letters |
| | will be sent to the authors soon. #CIKM2023 |
| | example_title: Unlabeled 3 |
| | - text: Ummmmmm |
| | example_title: Unlabeled 4 |
| | - text: >- |
| | whoever says that The Last Jedi is a good movie is lying or trolling |
| | everyone |
| | example_title: Unlabeled 5 |
| | - text: >- |
| | I don’t think people realize how big this story is GBI Strategies, the group |
| | paid $11M+ by Biden PACs to harvest fraudulent voter registrations in *20 |
| | states*, may be the root source of Democrat election rigging @USER may have |
| | just exposed their entire fraud machine HTTPURL |
| | example_tite: Unlabeled 6 |
| | base_model: |
| | - vinai/bertweet-base |
| | --- |
| | |
| | # TACO -- Twitter Arguments from COnversations |
| |
|
| | Introducing TACO, a baseline classification model built upon `AutoModelForSequenceClassification`, designed to identify tweets belonging to four distinct |
| | classes: Reason, Statement, Notification, and None. Tailored specifically for argument mining on Twitter, this baseline model is an evolution of the |
| | [BERTweet-base](https://huggingface.co/vinai/bertweet-base) architecture, which was originally pre-trained on Twitter data. |
| | Through fine-tuning with the [TACO dataset](https://github.com/TomatenMarc/TACO), the baseline model acquires its name and excels in the |
| | extraction of *Twitter Arguments from COnversations*. |
| |
|
| | ## Class Semantics |
| |
|
| | The TACO framework revolves around the two key elements of an argument, as defined by the [Cambridge Dictionary](https://dictionary.cambridge.org). |
| | It encodes *inference* as *a guess that you make or an opinion that you form based on the information that you have*, and it also leverages the |
| | definition of *information* as *facts or details about a person, company, product, etc.*. |
| |
|
| | Taken together, the following classes of tweets can be identified by TACO: |
| |
|
| | * *Statement*, which refers to unique cases where only the *inference* is presented as *something that someone says or writes officially, or an action |
| | done to express an opinion*. |
| | * *Reason*, which represents a full argument where the *inference* is based on direct *information* mentioned in the tweet, such as a source-reference |
| | or quotation, and thus reveals the author’s motivation *to try to understand and to make judgments based on practical facts*. |
| | * *Notification*, which refers to a tweet that limits itself to providing *information*, such as media channels promoting their latest articles. |
| | * *None*, a tweet that provides neither *inference* nor *information*. |
| |
|
| |
|
| | In its entirety, TACO can classify the following hierarchy for tweets: |
| |
|
| | <div align="center"> |
| | <img src="https://github.com/TomatenMarc/public-images/raw/main/Argument_Tree.svg" alt="Argument Tree" width="100%"> |
| | </div> |
| |
|
| | ## Usage |
| |
|
| | Using this model becomes easy when you have `transformers` installed: |
| |
|
| | ```python |
| | pip install - U transformers |
| | ``` |
| |
|
| | Then you can use the model to generate tweet classifications like this: |
| |
|
| | ```python |
| | from transformers import pipeline |
| | |
| | pipe = pipeline("text-classification", model="TomatenMarc/TACO") |
| | prediction = pipe("Huggingface is awesome") |
| | |
| | print(prediction) |
| | ``` |
| |
|
| | <a href="https://github.com/TomatenMarc/TACO/blob/main/notebooks/classifier_cv.ipynb"> |
| | <blockquote style="border-left: 5px solid grey; background-color: #f0f5ff; padding: 10px;"> |
| | Notice: The tweets need to undergo preprocessing before classification. |
| | </blockquote> |
| | </a> |
| | |
| | ## Training |
| |
|
| | The final model underwent training using the entire shuffled ground truth dataset known as TACO, encompassing a total of 1734 tweets. |
| | This dataset showcases the distribution of topics as: #abortion (25.9%), #brexit (29.0%), #got (11.0%), #lotrrop (12.1%), #squidgame (12.7%), and |
| | #twittertakeover (9.3%). For training, we utilized [SimpleTransformers](https://simpletransformers.ai). |
| |
|
| | Additionally, the category and class distribution of the dataset TACO is as follows: |
| |
|
| | | Argument | No-Argument | |
| | |--------------|------------------| |
| | | 865 (49.88%) | 869 (50.12%) | |
| |
|
| | | Reason | Statement | Notification | None | |
| | |--------------|--------------|--------------|--------------| |
| | | 581 (33.50%) | 284 (16.38%) | 500 (28.84%) | 369 (21.28%) | |
| |
|
| | <p> |
| | <blockquote style="border-left: 5px solid grey; background-color: #f0f5ff; padding: 10px;"> |
| | Notice: Our training involved TACO to forecast class predictions, where the categories (Argument/No-Argument) represent class aggregations |
| | based on the inference component. |
| | </blockquote> |
| | <p> |
| | |
| | ### Dataloader |
| |
|
| | ``` |
| | "data_loader": { |
| | "type": "torch.utils.data.dataloader.DataLoader", |
| | "args": { |
| | "batch_size": 8, |
| | "sampler": "torch.utils.data.sampler.RandomSampler" |
| | } |
| | } |
| | ``` |
| |
|
| | Parameters of the fit()-Method: |
| |
|
| | ``` |
| | { |
| | "epochs": 5, |
| | "max_grad_norm": 1, |
| | "optimizer_class": "<class 'torch.optim.adamw.AdamW'>", |
| | "optimizer_params": { |
| | "lr": 4e-05 |
| | }, |
| | "scheduler": "WarmupLinear", |
| | "warmup_steps": 66, |
| | "weight_decay": 0.06 |
| | } |
| | ``` |
| |
|
| | ## Evaluation |
| |
|
| | We utilized a stratified 10-fold cross-validation approach to present TACO's performance. In doing so, we employed the identical data and parameters |
| | as outlined in the *Training* section. This involved training on k-1 splits and utilizing the kth split for making predictions. |
| |
|
| | In total, the TACO classifier performs as follows: |
| |
|
| | ### Classification |
| |
|
| | | | Precision | Recall | F1-Score | Support | |
| | |-------------|-----------|---------|----------|---------| |
| | | Reason | 73.69% | 75.22% | 74.45% | 581 | |
| | | Statement | 54.37% | 59.15% | 56.66% | 284 | |
| | | Notification| 79.02% | 77.60% | 78.30% | 500 | |
| | | None | 83.87% | 77.51% | 80.56% | 369 | |
| | |-------------|-----------|---------|----------|---------| |
| | | Accuracy | | | 73.76% | 1734 | |
| | | Macro Avg | 72.74% | 72.37% | 72.49% | 1734 | |
| | | Weighted Avg| 74.23% | 73.76% | 73.95% | 1734 | |
| |
|
| | ### Categorization |
| |
|
| | | | Precision | Recall | F1-Score | Support | |
| | |-------------|-----------|---------|----------|---------| |
| | | No-Argument | 86.66% | 82.97% | 84.77% | 869 | |
| | | Argument | 83.59% | 87.17% | 85.34% | 865 | |
| | |-------------|-----------|---------|----------|---------| |
| | | Accuracy | | | 85.06% | 1734 | |
| | | Macro Avg | 85.13% | 85.07% | 85.06% | 1734 | |
| | | Weighted Avg| 85.13% | 85.06% | 85.06% | 1734 | |
| |
|
| | # Environmental Impact |
| |
|
| | - **Hardware Type:** A100 PCIe 40GB |
| | - **Hours used:** 10 min |
| | - **Cloud Provider:** [Google Cloud Platform](https://colab.research.google.com) |
| | - **Compute Region:** [asia-southeast1](https://cloud.google.com/compute/docs/gpus/gpu-regions-zones?hl=en) (Singapore) |
| | - **Carbon Emitted:** 0.02kg CO2 |
| |
|
| | # Licensing |
| |
|
| | [TACO](https://huggingface.co/TomatenMarc/TACO) © 2023 is licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/?ref=chooser-v1) |
| |
|
| |
|
| | # Citation |
| |
|
| | ``` |
| | @inproceedings{feger-dietze-2024-taco, |
| | title = "{TACO} {--} {T}witter Arguments from {CO}nversations", |
| | author = "Feger, Marc and |
| | Dietze, Stefan", |
| | editor = "Calzolari, Nicoletta and |
| | Kan, Min-Yen and |
| | Hoste, Veronique and |
| | Lenci, Alessandro and |
| | Sakti, Sakriani and |
| | Xue, Nianwen", |
| | booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)", |
| | month = may, |
| | year = "2024", |
| | address = "Torino, Italia", |
| | publisher = "ELRA and ICCL", |
| | url = "https://aclanthology.org/2024.lrec-main.1349", |
| | pages = "15522--15529" |
| | } |
| | ``` |
