KoModernBERT-large-mlm-v8

This model is a fine-tuned version of CocoRoF/KoModernBERT-large-mlm-v7 on an unknown dataset. It achieves the following results on the evaluation set:

Loss: 1.9913

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-06
train_batch_size: 4
eval_batch_size: 4
seed: 42
distributed_type: multi-GPU
num_devices: 8
gradient_accumulation_steps: 32
total_train_batch_size: 1024
total_eval_batch_size: 32
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.98) and epsilon=1e-07 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
70.789	0.0599	500	2.2607
73.2966	0.1199	1000	2.3530
72.0969	0.1798	1500	2.3316
73.4429	0.2397	2000	2.3031
72.2721	0.2997	2500	2.2628
70.2865	0.3596	3000	2.2342
70.2067	0.4196	3500	2.2033
68.5196	0.4795	4000	2.1734
66.5987	0.5394	4500	2.1468
66.429	0.5994	5000	2.1171
65.128	0.6593	5500	2.0887
65.0296	0.7192	6000	2.0684
64.7926	0.7792	6500	2.0442
64.4718	0.8391	7000	2.0191
63.3394	0.8990	7500	2.0055
62.7926	0.9590	8000	1.9913