README.md

metadata

library_name: transformers
license: apache-2.0
base_model: x2bee/KoModernBERT-base-mlm-v03-retry-ckp02
tags:
  - generated_from_trainer
model-index:
  - name: KMB_SimCSE_test
    results: []

KMB_SimCSE_test

This model is a fine-tuned version of x2bee/KoModernBERT-base-mlm-v03-retry-ckp02 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0463
• Pearson Cosine: 0.8264
• Spearman Cosine: 0.8281
• Pearson Manhattan: 0.8304
• Spearman Manhattan: 0.8368
• Pearson Euclidean: 0.8298
• Spearman Euclidean: 0.8363
• Pearson Dot: 0.7651
• Spearman Dot: 0.7620

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• distributed_type: multi-GPU
• gradient_accumulation_steps: 8
• total_train_batch_size: 128
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss	Pearson Cosine	Spearman Cosine	Pearson Manhattan	Spearman Manhattan	Pearson Euclidean	Spearman Euclidean	Pearson Dot	Spearman Dot
0.6339	0.0469	100	0.0942	0.7833	0.7817	0.7822	0.7875	0.7816	0.7865	0.7100	0.7029
0.448	0.0937	200	0.0808	0.7856	0.7882	0.7887	0.7945	0.7873	0.7930	0.7399	0.7354
0.427	0.1406	300	0.0699	0.8037	0.8029	0.8021	0.8090	0.8010	0.8077	0.7505	0.7461
0.4052	0.1874	400	0.0616	0.8109	0.8096	0.8094	0.8143	0.8080	0.8128	0.7663	0.7620
0.4023	0.2343	500	0.0612	0.8109	0.8135	0.8133	0.8187	0.8113	0.8168	0.7739	0.7705
0.3754	0.2812	600	0.0595	0.8105	0.8125	0.8101	0.8162	0.8087	0.8146	0.7706	0.7681
0.3729	0.3280	700	0.0619	0.8155	0.8176	0.8144	0.8212	0.8128	0.8196	0.7761	0.7736
0.341	0.3749	800	0.0530	0.8137	0.8155	0.8201	0.8246	0.8190	0.8234	0.7696	0.7663
0.3161	0.4217	900	0.0568	0.8162	0.8182	0.8209	0.8262	0.8198	0.8252	0.7660	0.7625
0.3122	0.4686	1000	0.0541	0.8215	0.8236	0.8220	0.8284	0.8207	0.8268	0.7790	0.7745
0.3301	0.5155	1100	0.0617	0.8116	0.8150	0.8177	0.8228	0.8161	0.8212	0.7638	0.7598
0.3637	0.5623	1200	0.0532	0.8108	0.8145	0.8175	0.8222	0.8156	0.8202	0.7681	0.7643
0.2885	0.6092	1300	0.0451	0.8272	0.8278	0.8275	0.8324	0.8268	0.8318	0.7925	0.7888
0.2852	0.6560	1400	0.0473	0.8246	0.8264	0.8228	0.8281	0.8221	0.8275	0.7893	0.7874
0.3225	0.7029	1500	0.0507	0.8259	0.8284	0.8274	0.8335	0.8263	0.8325	0.7737	0.7708
0.3201	0.7498	1600	0.0467	0.8248	0.8268	0.8232	0.8282	0.8222	0.8274	0.7800	0.7772
0.3199	0.7966	1700	0.0511	0.8215	0.8239	0.8266	0.8322	0.8257	0.8308	0.7702	0.7658
0.2431	0.8435	1800	0.0482	0.8271	0.8287	0.8282	0.8333	0.8277	0.8326	0.7791	0.7749
0.3051	0.8903	1900	0.0465	0.8277	0.8295	0.8257	0.8324	0.8249	0.8319	0.7814	0.7782
0.3287	0.9372	2000	0.0551	0.8207	0.8244	0.8238	0.8296	0.8229	0.8287	0.7620	0.7569
0.2889	0.9841	2100	0.0463	0.8264	0.8281	0.8304	0.8368	0.8298	0.8363	0.7651	0.7620

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu121
• Datasets 3.1.0
• Tokenizers 0.21.0