PerRing/e5_ko

SentenceTransformer based on intfloat/multilingual-e5-large

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/multilingual-e5-large
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("PerRing/e5_ko")
# Run inference
sentences = [
    'query: 그의 사망 약 30년 후에 극장들은 파괴되었다.',
    'passage: 그가 사망한 지 약 30년 후에 극장들이 무너졌다.',
    'passage: 그의 사망 약 30년 후에 극장들은 번창했다.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 49,962 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 8 tokens mean: 20.11 tokens max: 68 tokens	min: 7 tokens mean: 19.17 tokens max: 62 tokens	min: 7 tokens mean: 18.47 tokens max: 48 tokens

• Samples:

  anchor	positive	negative
  query: 사람들이 기억하면 당신은 다음 수준으로 물건을 잃게 됩니다.	passage: 사람들이 상기하면 당신은 다음 단계로 물건을 잃습니다.	passage: 사람들이 기억하지 않으면 당신은 물건을 잃지 않고 유지합니다.
  query: 제 팀의 한 멤버가 당신의 명령을 매우 정확하게 수행할 것입니다.	passage: 제 팀원 중 한 명이 당신의 지시를 철저히 이행할 것입니다.	passage: 제 팀의 누구도 당신의 명령을 정확히 수행하지 않을 것입니다.
  query: 이 정보는 그들의 것입니다.	passage: 이 정보는 그들 소유입니다.	passage: 이 정보는 우리 것입니다.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• num_train_epochs: 2
• warmup_ratio: 0.1
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss
0.04	1	5.7804
0.08	2	5.7809
0.12	3	4.8376
0.16	4	3.6166
0.2	5	2.4258
0.24	6	1.2571
0.28	7	0.7152
0.32	8	0.5181
0.36	9	0.359
0.4	10	0.3295
0.44	11	0.2502
0.48	12	0.2256
0.52	13	0.1842
0.56	14	0.1588
0.6	15	0.1614
0.64	16	0.1535
0.68	17	0.14
0.72	18	0.152
0.76	19	0.1221
0.8	20	0.1179
0.84	21	0.1354
0.88	22	0.1349
0.92	23	0.1364
0.96	24	0.1169
1.0	25	0.0863
1.04	26	0.1161
1.08	27	0.1105
1.12	28	0.1067
1.16	29	0.1087
1.2	30	0.1165
1.24	31	0.1082
1.28	32	0.0866
1.32	33	0.0854
1.3600	34	0.0905
1.4	35	0.0809
1.44	36	0.079
1.48	37	0.0665
1.52	38	0.0542
1.56	39	0.06
1.6	40	0.0624
1.6400	41	0.0671
1.6800	42	0.0721
1.72	43	0.0781
1.76	44	0.0531
1.8	45	0.0556
1.8400	46	0.0732
1.88	47	0.0748
1.92	48	0.0773
1.96	49	0.068
2.0	50	0.0347

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.2.1
• Transformers: 4.45.2
• PyTorch: 2.4.1+cu121
• Accelerate: 1.1.0
• Datasets: 3.1.0
• Tokenizers: 0.20.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}