ESIEA-NLP/fine-tuned-RoBERTa

Fine Tuned RoBERTa model

This model is a fine-tuned version of roberta-base on the Google Jigsaw dataset. It achieves the following results on the evaluation set:

• Loss: 0.0402
• F1: 0.7890
• Roc Auc: 0.8997
• Accuracy: 0.9245

Model description

This model is specifically designed to classify the toxicity of comments. It is based on the RoBERTa architecture and has been fine-tuned on the Google Jigsaw dataset. The model has been trained to predict among the following labels : toxic, severe toxic, obscene, threat, insult, and identity hate. Note that a comment can have multiple labels.

Intended uses & limitations

This model was fully developed during a student project and is intended for educational purposes only.

Training and evaluation data

The model was trained on the Google Jigsaw dataset. We have about 150 000 training samples and about 150 000 test samples. In test sample, not all samples have labels.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 13

Training results

Training Loss	Epoch	Step	Validation Loss	F1	Roc Auc	Accuracy
0.0415	1.0	3491	0.0448	0.7697	0.8990	0.9199
0.0348	2.0	6982	0.0402	0.7890	0.8997	0.9245
0.0345	3.0	10473	0.0414	0.7880	0.8880	0.9251
0.036	4.0	13964	0.0448	0.7850	0.8990	0.9219
0.0199	5.0	17455	0.0471	0.7882	0.8848	0.9256
0.0216	6.0	20946	0.0522	0.7803	0.8830	0.9241
0.0169	7.0	24437	0.0601	0.7782	0.8912	0.9211
0.0134	8.0	27928	0.0651	0.7777	0.8955	0.9197
0.0069	9.0	31419	0.0709	0.7690	0.8992	0.9159
0.0085	10.0	34910	0.0744	0.7776	0.8848	0.9218
0.0012	11.0	38401	0.0783	0.7768	0.8828	0.9217
0.0028	12.0	41892	0.0804	0.7752	0.8891	0.9202
0.005	13.0	45383	0.0817	0.7778	0.8925	0.9202

Framework versions

• Transformers 4.46.3
• Pytorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.20.3