sarcasm-detector

This model is a fine-tuned version of distilbert/distilbert-base-uncased on [raquiba/Sarcasm_News_Headline]{https://huggingface.co/datasets/raquiba/Sarcasm_News_Headline} dataset. It achieves the following results on the evaluation set:

• Loss: 0.0279
• Accuracy: 0.9937

Model description

This model was trained for Sarcasm Detection in customer feedbacks for businesses in the e-commerce, banking, and telecommunication sectors. The base model used for this is distilbert/distilbert-base-uncased and it was trained on the [raquiba/Sarcasm_News_Headline]{https://huggingface.co/datasets/raquiba/Sarcasm_News_Headline} dataset. It is intended to be used in pipeline with multiple models so that the result generated by this model can be used as parameters for other models, this feature couldn't be implemented as even though the accuracy that was calculated was high, the model still labels the text randomly for unknown reason.

Possible Reasons for wrong labelling done by the Sarcasm Model:

• Overfitting: Instead of identifying patterns in the dataset, the model memorized the dataset itself, this lead to another problem Hyper-sensitivity.
• The dataset basically has news headlines, headlines are obviously small so in order to convey sarcasm most news headlines use very expressive words like 'Wow! another robbery, how shocking'.
• The model learned this and became hyper-sensitive to such words and phrases, so instead of detecting sarcasm the model started looking for these words to flag the sentence with 'SARCASM' label.

For example: "Our team successfully migrated all legacy database infrastructure to your cloud platform over the weekend without experiencing a single second of unexpected system downtime. The automated data synchronization tools functioned seamlessly exactly as advertised, saving our engineering department dozens of hours of manual validation that we had originally budgeted for." This sentece contains genuine praise but since it uses high level of expressing to depict appreciation, it was flagged as 'SARCASM' by the model.

Intended uses & limitations

The model was intended to be used as a Sarcasm Detector in a multi-model pipeline alongside other models for sentiment analysis, emotion detection, urgency detetion, and category classification. This model was supposed to help with creating sarcasm-aware sentiment analysis model, urgency detection model, and emotion detection model; as models trained for these tasks individually often lack the ability to detect human sarcasm. The sarcasm model could have provided a flag value to the other models, that could take it under consideration while making their own predictions, making those models sarcasm-aware. This feature couldn't be implemented due to limitation like:

• Lack of a thorough dataset, as human sarcasm can have many forms.
• Hardware limitation prevented the use of a better base model.
• A synthetic dataset could have been created using Local LLMs like Llama, but that was also prevented due to hardware limitations
• Due to time-constraint the model couldn't be repeatedly developed and tested.
• The model suffered through a serious case of over-fitting and hyper-sensitivity.

Training and evaluation data

The training and evaluation data comes from the dataset used, from the 'train' and 'test' split of the dataset, which was passed to the 'Trainer()' as arguments:

• train_dataset=tokenized_news["train"],
• eval_dataset=tokenized_news["test"],

Training procedure

The model was fine-tuned using the Hugging Face 'Trainer' API.

• The dataset was loaded using the 'load_dataset()'' function from the 'datasets' library.
• The dataset went through processing: removing the unnecessary columns and renaming the column storing the labels for sarcasm.
• The tokenizer was initialized using the 'AutoTokenizer.from_pretrained()' from the Transformers Library.
• The dataset was tokenized using the tokenizer
• The data collator was initialized using the 'DataCollatorWithPadding()' from the 'transformers' Library.
• The evaluation metrics was initialized as 'accuracy' using the 'load()' function from the 'evaluate' Library.
• The pre-trained model was loaded using the 'AutoModelForSequenceClassification.from_pretrained()' from the 'transformers' Library.
• The training arguments were initialized using the 'TrainingArguments()' function from the 'transformers' Library with the Hyperparameters listed below.
• The trainer was initialized using the 'Trainer()' method from the 'transformers' Library.

Training hyperparameters

The following hyperparameters were used during training: - output_dir="sarcasm-detector" - learning_rate: 2e-05 - train_batch_size: 16 - eval_batch_size: 16 - num_train_epochs=3 - weight_decay=0.01 - eval_strategy="epoch" - save_strategy="epoch" - load_best_model_at_end=True - seed: 42 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.2293	1.0	1789	0.1074	0.9610
0.1312	2.0	3578	0.0417	0.9883
0.0615	3.0	5367	0.0279	0.9937

Framework versions

• Transformers 5.0.0
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2