ModernBERT-domain-classifier

This model is a fine-tuned version of answerdotai/ModernBERT-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7495

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 4
• eval_batch_size: 2
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
1.1514	0.0786	500	1.1314
1.1393	0.1573	1000	1.1216
1.1294	0.2359	1500	1.0559
1.0057	0.3145	2000	0.8818
0.8767	0.3931	2500	0.7991
0.8361	0.4718	3000	0.8994
0.7708	0.5504	3500	0.7846
0.7679	0.6290	4000	0.6797
0.7113	0.7077	4500	0.7602
0.7513	0.7863	5000	0.7201
0.6913	0.8649	5500	0.7038
0.682	0.9435	6000	0.7809
0.6802	1.0222	6500	0.6597
0.5764	1.1008	7000	0.8693
0.6126	1.1794	7500	0.8703
0.6099	1.2581	8000	0.7533
0.6095	1.3367	8500	0.7253
0.5707	1.4153	9000	0.8022
0.5788	1.4939	9500	0.7313
0.5709	1.5726	10000	0.7223
0.5772	1.6512	10500	0.6996
0.5397	1.7298	11000	0.7453
0.5925	1.8085	11500	0.6742
0.5342	1.8871	12000	0.7517
0.581	1.9657	12500	0.7495

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.6.0+cu124
• Datasets 3.1.0
• Tokenizers 0.21.1