distilbert_add_GLUE_Experiment_logit_kd_stsb_96

This model is a fine-tuned version of distilbert-base-uncased on the GLUE STSB dataset. It achieves the following results on the evaluation set:

• Loss: 1.1264
• Pearson: nan
• Spearmanr: nan
• Combined Score: nan

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 256
• eval_batch_size: 256
• seed: 10
• distributed_type: multi-GPU
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Pearson	Spearmanr	Combined Score
4.3296	1.0	23	3.3387	nan	nan	nan
3.9535	2.0	46	3.1277	nan	nan	nan
3.7081	3.0	69	2.9189	nan	nan	nan
3.4597	4.0	92	2.7125	nan	nan	nan
3.2232	5.0	115	2.5114	nan	nan	nan
2.972	6.0	138	2.3156	0.0070	0.0078	0.0074
2.7373	7.0	161	2.1284	nan	nan	nan
2.527	8.0	184	1.9503	nan	nan	nan
2.3016	9.0	207	1.7828	0.0092	0.0081	0.0087
2.0903	10.0	230	1.6295	nan	nan	nan
1.8919	11.0	253	1.4932	-0.0357	-0.0358	-0.0358
1.7184	12.0	276	1.3768	nan	nan	nan
1.5665	13.0	299	1.2813	0.0302	0.0292	0.0297
1.4283	14.0	322	1.2075	0.0115	0.0132	0.0123
1.3175	15.0	345	1.1569	nan	nan	nan
1.2276	16.0	368	1.1298	nan	nan	nan
1.1643	17.0	391	1.1264	nan	nan	nan
1.1172	18.0	414	1.1447	0.0009	0.0027	0.0018
1.1066	19.0	437	1.1677	nan	nan	nan
1.1002	20.0	460	1.1712	0.0024	0.0003	0.0014
1.1027	21.0	483	1.1767	nan	nan	nan
1.0984	22.0	506	1.1799	nan	nan	nan

Framework versions

• Transformers 4.26.0
• Pytorch 1.14.0a0+410ce96
• Datasets 2.9.0
• Tokenizers 0.13.2