SOMD-xlm-stage2-v3

This model is a fine-tuned version of xlm-roberta-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0125

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 25

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	202	0.8204
No log	1.99	404	0.4907
0.8995	2.99	606	0.3113
0.8995	3.98	808	0.2219
0.3172	4.98	1010	0.1731
0.3172	5.97	1212	0.1251
0.3172	6.97	1414	0.0677
0.1475	7.96	1616	0.0648
0.1475	8.96	1818	0.0422
0.0761	9.95	2020	0.0406
0.0761	10.95	2222	0.0280
0.0761	11.94	2424	0.0305
0.0484	12.94	2626	0.0251
0.0484	13.93	2828	0.0254
0.0382	14.93	3030	0.0207
0.0382	15.92	3232	0.0223
0.0382	16.92	3434	0.0168
0.0287	17.91	3636	0.0191
0.0287	18.91	3838	0.0158
0.0263	19.9	4040	0.0149
0.0263	20.9	4242	0.0155
0.0263	21.89	4444	0.0138
0.0228	22.89	4646	0.0130
0.0228	23.88	4848	0.0131
0.018	24.88	5050	0.0125

Framework versions

• Transformers 4.37.2
• Pytorch 2.1.0+cu121
• Datasets 2.17.1
• Tokenizers 0.15.2