bert-base-uncased-FinedTuned

This model is a fine-tuned version of bert-base-uncased on the stsb_multi_mt dataset. It achieves the following results on the evaluation set:

• Loss: 2.7758
• Pearson: 0.2352
• Mse: 2.7758
• Custom Accuracy: 0.2611
• Dataset Accuracy: 0.1762

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 1000
• training_steps: 12000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Pearson	Mse	Custom Accuracy	Dataset Accuracy
0.028	5.5556	1000	2.7386	0.2467	2.7386	0.2502	0.1762
0.0269	11.1111	2000	2.8265	0.2229	2.8265	0.2589	0.1762
0.0088	16.6667	3000	2.8485	0.2219	2.8485	0.2654	0.1762
0.0141	22.2222	4000	2.8855	0.2086	2.8855	0.2661	0.1762
0.0099	27.7778	5000	2.8081	0.2328	2.8081	0.2632	0.1762
0.0248	33.3333	6000	2.7765	0.2309	2.7765	0.2625	0.1762
0.0353	38.8889	7000	2.8126	0.2296	2.8126	0.2748	0.1762
0.0892	44.4444	8000	2.8362	0.2327	2.8362	0.2567	0.1762
0.0488	50.0	9000	2.7667	0.2363	2.7667	0.2596	0.1762
0.0538	55.5556	10000	2.7885	0.2363	2.7885	0.2632	0.1762
0.0829	61.1111	11000	2.7837	0.2348	2.7837	0.2647	0.1762
0.1473	66.6667	12000	2.7758	0.2352	2.7758	0.2611	0.1762

Framework versions

• Transformers 4.42.3
• Pytorch 2.3.1+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1