roberta-base-multi-head-eval-loss-600-steps

This model is a fine-tuned version of roberta-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.6123
• Accuracy: 0.5454
• F1 Macro: 0.5277
• F1 Micro: 0.5454
• Precision Macro: 0.5265
• Recall Macro: 0.5355
• Roc Auc: 0.7682

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.05
• num_epochs: 40

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1 Macro	F1 Micro	Precision Macro	Recall Macro	Roc Auc
0.6426	0.3913	600	0.5641	0.4010	0.2319	0.4010	0.2123	0.2946	0.5958
0.553	0.7826	1200	0.4983	0.5054	0.4098	0.5054	0.4788	0.4292	0.7266
0.5092	1.1735	1800	0.4789	0.5307	0.4701	0.5307	0.4958	0.4705	0.7549
0.4813	1.5648	2400	0.4631	0.5553	0.5108	0.5553	0.5302	0.5105	0.7751
0.4635	1.9561	3000	0.4667	0.5432	0.5188	0.5432	0.5230	0.5255	0.7786
0.4399	2.3469	3600	0.4634	0.5570	0.5203	0.5570	0.5495	0.5267	0.7827
0.429	2.7382	4200	0.4689	0.5403	0.5159	0.5403	0.5361	0.5304	0.7873
0.4332	3.1291	4800	0.4613	0.5625	0.5348	0.5625	0.5429	0.5390	0.7894
0.4032	3.5204	5400	0.4735	0.5597	0.5392	0.5597	0.5381	0.5459	0.7857
0.3994	3.9117	6000	0.4733	0.5577	0.5331	0.5577	0.5440	0.5414	0.7870
0.37	4.3026	6600	0.4799	0.5554	0.5344	0.5554	0.5343	0.5387	0.7862
0.3662	4.6939	7200	0.4882	0.5635	0.5406	0.5635	0.5452	0.5448	0.7862
0.3588	5.0848	7800	0.5039	0.5557	0.5319	0.5557	0.5397	0.5408	0.7833
0.339	5.4761	8400	0.5172	0.5504	0.5300	0.5504	0.5324	0.5384	0.7809
0.3298	5.8674	9000	0.5036	0.5683	0.5389	0.5683	0.5497	0.5369	0.7820
0.3038	6.2583	9600	0.5511	0.5524	0.5217	0.5524	0.5439	0.5224	0.7738
0.2906	6.6495	10200	0.5465	0.5564	0.5300	0.5564	0.5388	0.5295	0.7745
0.2911	7.0404	10800	0.5771	0.5529	0.5323	0.5529	0.5310	0.5397	0.7744
0.2825	7.4317	11400	0.5929	0.5464	0.5202	0.5464	0.5293	0.5263	0.7684
0.2577	7.8230	12000	0.5975	0.5520	0.5324	0.5520	0.5299	0.5355	0.7686
0.2442	8.2139	12600	0.6123	0.5454	0.5277	0.5454	0.5265	0.5355	0.7682

Framework versions

• Transformers 4.53.1
• Pytorch 2.6.0+cu124
• Datasets 2.14.4
• Tokenizers 0.21.2