sa_BERT_48_stsb

This model is a fine-tuned version of gokuls/bert_base_48 on the GLUE STSB dataset. It achieves the following results on the evaluation set:

• Loss: 1.8552
• Pearson: 0.5082
• Spearmanr: 0.4883
• Combined Score: 0.4983

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 4e-05
• train_batch_size: 96
• eval_batch_size: 96
• seed: 10
• distributed_type: multi-GPU
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 50

Training results

Training Loss	Epoch	Step	Validation Loss	Pearson	Spearmanr	Combined Score
2.185	1.0	60	2.8052	0.1411	0.1236	0.1323
1.8984	2.0	120	2.3141	0.1796	0.1681	0.1738
1.5809	3.0	180	2.2961	0.2130	0.2067	0.2099
1.2083	4.0	240	2.6953	0.2673	0.2669	0.2671
0.8831	5.0	300	2.7535	0.2934	0.2785	0.2859
0.6722	6.0	360	2.5880	0.3524	0.3452	0.3488
0.5242	7.0	420	3.2023	0.3644	0.3526	0.3585
0.4244	8.0	480	2.2353	0.4509	0.4368	0.4438
0.3482	9.0	540	2.1408	0.4592	0.4435	0.4514
0.2885	10.0	600	2.1727	0.4457	0.4343	0.4400
0.2641	11.0	660	2.2167	0.4599	0.4462	0.4531
0.2319	12.0	720	2.2479	0.4522	0.4296	0.4409
0.1991	13.0	780	2.0491	0.4659	0.4495	0.4577
0.1791	14.0	840	2.0965	0.4770	0.4556	0.4663
0.1692	15.0	900	2.2466	0.4860	0.4721	0.4790
0.1406	16.0	960	2.0543	0.4980	0.4790	0.4885
0.146	17.0	1020	1.9725	0.4874	0.4697	0.4785
0.1205	18.0	1080	2.1711	0.4560	0.4347	0.4454
0.123	19.0	1140	2.1570	0.4622	0.4394	0.4508
0.1075	20.0	1200	2.0031	0.4652	0.4452	0.4552
0.107	21.0	1260	2.1680	0.4588	0.4302	0.4445
0.0903	22.0	1320	1.8876	0.4938	0.4705	0.4821
0.0866	23.0	1380	1.8552	0.5082	0.4883	0.4983
0.085	24.0	1440	2.1126	0.4735	0.4489	0.4612
0.0786	25.0	1500	1.9673	0.4998	0.4772	0.4885
0.0713	26.0	1560	1.9474	0.4897	0.4680	0.4788
0.0715	27.0	1620	1.9366	0.5000	0.4795	0.4897
0.0612	28.0	1680	2.0001	0.4809	0.4538	0.4673

Framework versions

• Transformers 4.30.2
• Pytorch 1.14.0a0+410ce96
• Datasets 2.13.0
• Tokenizers 0.13.3