distilbert_sa_GLUE_Experiment_mrpc_96

This model is a fine-tuned version of distilbert-base-uncased on the GLUE MRPC dataset. It achieves the following results on the evaluation set:

• Loss: 0.5873
• Accuracy: 0.6887
• F1: 0.7829
• Combined Score: 0.7358

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 256
• eval_batch_size: 256
• seed: 10
• distributed_type: multi-GPU
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Combined Score
0.6677	1.0	15	0.6479	0.6838	0.8122	0.7480
0.6455	2.0	30	0.6395	0.6838	0.8122	0.7480
0.6399	3.0	45	0.6331	0.6838	0.8122	0.7480
0.6361	4.0	60	0.6288	0.6838	0.8122	0.7480
0.6352	5.0	75	0.6262	0.6838	0.8122	0.7480
0.6315	6.0	90	0.6252	0.6838	0.8122	0.7480
0.6331	7.0	105	0.6244	0.6838	0.8122	0.7480
0.6292	8.0	120	0.6242	0.6838	0.8122	0.7480
0.6314	9.0	135	0.6240	0.6838	0.8122	0.7480
0.6296	10.0	150	0.6242	0.6838	0.8122	0.7480
0.6306	11.0	165	0.6241	0.6838	0.8122	0.7480
0.63	12.0	180	0.6240	0.6838	0.8122	0.7480
0.6337	13.0	195	0.6240	0.6838	0.8122	0.7480
0.6299	14.0	210	0.6239	0.6838	0.8122	0.7480
0.6297	15.0	225	0.6230	0.6838	0.8122	0.7480
0.6248	16.0	240	0.6187	0.6838	0.8122	0.7480
0.6065	17.0	255	0.5999	0.6936	0.8164	0.7550
0.5624	18.0	270	0.6007	0.6838	0.7659	0.7249
0.5185	19.0	285	0.5891	0.6838	0.7772	0.7305
0.4664	20.0	300	0.5873	0.6887	0.7829	0.7358
0.4248	21.0	315	0.5893	0.6936	0.7764	0.7350
0.3844	22.0	330	0.5949	0.7010	0.7798	0.7404
0.3551	23.0	345	0.5942	0.7034	0.7866	0.7450
0.3314	24.0	360	0.6040	0.7034	0.7881	0.7458
0.3181	25.0	375	0.6162	0.7010	0.7867	0.7438

Framework versions

• Transformers 4.26.0
• Pytorch 1.14.0a0+410ce96
• Datasets 2.8.0
• Tokenizers 0.13.2