license: mit
tags:
- generated_from_trainer
model-index:
- name: Clickbait1
results: []
Clickbait1
This model is a fine-tuned version of microsoft/Multilingual-MiniLM-L12-H384 on the Webis-Clickbait-17 dataset. It achieves the following results on the evaluation set:
- Loss: 0.0260
Model description
MiniLM is a distilled model from the paper "MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers".
We fine tune this model to evaluate (regression) the clickbait level of title news.
Intended uses & limitations
Model was designed to work on Transformers (like in the paper "Predicting Clickbait Strength in Online Social Media" by Indurthi Vijayasaradhi, Syed Bakhtiyar, Gupta Manish, Varma Vasudeva).
The model wa trained with english titles.
Training and evaluation data
We train the model with the official training data of the chalenge, plus another set that was available after the end of the challenge.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 4
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| No log | 0.05 | 50 | 0.0390 |
| No log | 0.09 | 100 | 0.0365 |
| No log | 0.14 | 150 | 0.0321 |
| No log | 0.18 | 200 | 0.0319 |
| No log | 0.23 | 250 | 0.0350 |
| No log | 0.27 | 300 | 0.0317 |
| No log | 0.32 | 350 | 0.0304 |
| No log | 0.36 | 400 | 0.0283 |
| No log | 0.41 | 450 | 0.0300 |
| 0.0364 | 0.46 | 500 | 0.0282 |
| 0.0364 | 0.5 | 550 | 0.0285 |
| 0.0364 | 0.55 | 600 | 0.0276 |
| 0.0364 | 0.59 | 650 | 0.0278 |
| 0.0364 | 0.64 | 700 | 0.0293 |
| 0.0364 | 0.68 | 750 | 0.0280 |
| 0.0364 | 0.73 | 800 | 0.0320 |
| 0.0364 | 0.77 | 850 | 0.0269 |
| 0.0364 | 0.82 | 900 | 0.0269 |
| 0.0364 | 0.87 | 950 | 0.0271 |
| 0.0281 | 0.91 | 1000 | 0.0314 |
| 0.0281 | 0.96 | 1050 | 0.0265 |
| 0.0281 | 1.0 | 1100 | 0.0295 |
| 0.0281 | 1.05 | 1150 | 0.0295 |
| 0.0281 | 1.09 | 1200 | 0.0290 |
| 0.0281 | 1.14 | 1250 | 0.0281 |
| 0.0281 | 1.18 | 1300 | 0.0272 |
| 0.0281 | 1.23 | 1350 | 0.0273 |
| 0.0281 | 1.28 | 1400 | 0.0287 |
| 0.0281 | 1.32 | 1450 | 0.0267 |
| 0.026 | 1.37 | 1500 | 0.0284 |
| 0.026 | 1.41 | 1550 | 0.0264 |
| 0.026 | 1.46 | 1600 | 0.0273 |
| 0.026 | 1.5 | 1650 | 0.0280 |
| 0.026 | 1.55 | 1700 | 0.0266 |
| 0.026 | 1.59 | 1750 | 0.0260 |
| 0.026 | 1.64 | 1800 | 0.0266 |
| 0.026 | 1.68 | 1850 | 0.0268 |
| 0.026 | 1.73 | 1900 | 0.0269 |
| 0.026 | 1.78 | 1950 | 0.0260 |
| 0.0236 | 1.82 | 2000 | 0.0273 |
| 0.0236 | 1.87 | 2050 | 0.0272 |
| 0.0236 | 1.91 | 2100 | 0.0260 |
| 0.0236 | 1.96 | 2150 | 0.0269 |
| 0.0236 | 2.0 | 2200 | 0.0286 |
| 0.0236 | 2.05 | 2250 | 0.0266 |
Framework versions
- Transformers 4.18.0
- Pytorch 1.11.0a0+17540c5
- Datasets 2.1.0
- Tokenizers 0.12.1