tiny-mlm-glue-wnli

This model is a fine-tuned version of google/bert_uncased_L-2_H-128_A-2 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.7902

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• num_epochs: 200

Training results

Training Loss	Epoch	Step	Validation Loss
3.5021	6.25	500	2.8676
2.9602	12.5	1000	2.5328
2.6906	18.75	1500	2.3640
2.4167	25.0	2000	2.0918
2.2699	31.25	2500	2.1182
2.0933	37.5	3000	1.8802
1.9399	43.75	3500	1.8979
1.7961	50.0	4000	1.5276
1.6738	56.25	4500	1.5327
1.5784	62.5	5000	1.2767
1.4405	68.75	5500	1.3593
1.3428	75.0	6000	0.9772
1.3257	81.25	6500	1.2789
1.1988	87.5	7000	0.9494
1.1275	93.75	7500	0.8278
1.0854	100.0	8000	0.6301
1.0275	106.25	8500	0.8044
0.9714	112.5	9000	0.7544
0.9019	118.75	9500	0.7902

Framework versions

• Transformers 4.26.0.dev0
• Pytorch 1.13.0+cu116
• Datasets 2.8.1.dev0
• Tokenizers 0.13.2