2ADT-Consulting/susu-sentence-encoder-v1

SentenceTransformer based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Les navires de Hiram, qui apportèrent de lor dOphir, amenèrent aussi dOphir une grande quantité de bois de sandal et des pierres précieuses.',
    'Xirami nun Sulemani xa walikɛe naxee fa xɛɛma ra kelife Ofiri bɔxi ma, nee man naxa fa wuri nun gɛmɛ tofanyie ra.',
    'Won a kolon won nun Ala na a ra, barima a bara a Xaxili fi won ma.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6592, 0.1617],
#         [0.6592, 1.0000, 0.0717],
#         [0.1617, 0.0717, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 225,516 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 13 tokens mean: 38.24 tokens max: 93 tokens	min: 10 tokens mean: 50.44 tokens max: 115 tokens

• Samples:

  anchor	positive
  He made it according to the instructions that Moses, the Lord 's servant, had given the Israelites, as it says in the Law of Moses: an altar made of stones which have not been cut with iron tools. On it they offered burnt sacrifices to the Lord , and they also presented their fellowship offerings.	alɔ Alatala xa konyi Munsa a fala Isirayilakae bɛ ki naxɛ. E naxa sɛrɛxɛbade gɛmɛ daaxi ti, wure mu nu din naxan na, alɔ a sɛbɛxi Tawureta Munsa kitaabui kui ki naxɛ. E naxa sɛrɛxɛ gan daaxie nun xanunteya sɛrɛxɛe ba Alatala bɛ na fari.
  Celle-ci ordonna à Hathac de rapporter sa réponse à Mardochée :.	Esita to na mɛ, a man naxa Hataki xɛɛ Morodekayi xɔn ma, a xa sa yi fala a bɛ.
  But there is something you do that is right—you hate the things that the Nicolaitans do. I also hate what they do.	«Kɔnɔ wo bara fe nde raba naxan fan. Wo bara Nikolasi xa ɲama xa fe xɔn, alɔ n fan a xɔnxi ki naxɛ.».

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• learning_rate: 2e-05
• num_train_epochs: 4
• warmup_steps: 0.1
• fp16: True

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0284	50	5.0351
0.0568	100	4.5387
0.0851	150	4.0088
0.1135	200	3.2828
0.1419	250	2.6326
0.1703	300	2.0852
0.1986	350	1.6870
0.2270	400	1.4379
0.2554	450	1.2296
0.2838	500	1.0465
0.3121	550	0.8908
0.3405	600	0.8758
0.3689	650	0.7199
0.3973	700	0.6701
0.4257	750	0.6193
0.4540	800	0.5435
0.4824	850	0.5279
0.5108	900	0.4810
0.5392	950	0.4594
0.5675	1000	0.4344
0.5959	1050	0.3961
0.6243	1100	0.3767
0.6527	1150	0.3601
0.6810	1200	0.3690
0.7094	1250	0.3448
0.7378	1300	0.3396
0.7662	1350	0.3242
0.7946	1400	0.3236
0.8229	1450	0.2961
0.8513	1500	0.2853
0.8797	1550	0.2933
0.9081	1600	0.2625
0.9364	1650	0.2758
0.9648	1700	0.2617
0.9932	1750	0.2720
1.0216	1800	0.2535
1.0499	1850	0.2366
1.0783	1900	0.2097
1.1067	1950	0.2183
1.1351	2000	0.2201
1.1635	2050	0.2284
1.1918	2100	0.2259
1.2202	2150	0.2125
1.2486	2200	0.2059
1.2770	2250	0.1950
1.3053	2300	0.2066
1.3337	2350	0.1944
1.3621	2400	0.2019
1.3905	2450	0.2051
1.4188	2500	0.1903
1.4472	2550	0.1958
1.4756	2600	0.1869
1.5040	2650	0.1827
1.5323	2700	0.1804
1.5607	2750	0.1692
1.5891	2800	0.2033
1.6175	2850	0.1740
1.6459	2900	0.1810
1.6742	2950	0.1785
1.7026	3000	0.1737
1.7310	3050	0.1914
1.7594	3100	0.1779
1.7877	3150	0.1670
1.8161	3200	0.1744
1.8445	3250	0.1647
1.8729	3300	0.1720
1.9012	3350	0.1746
1.9296	3400	0.1559
1.9580	3450	0.1571
1.9864	3500	0.1655
2.0148	3550	0.1342
2.0431	3600	0.1304
2.0715	3650	0.1335
2.0999	3700	0.1290
2.1283	3750	0.1486
2.1566	3800	0.1274
2.1850	3850	0.1384
2.2134	3900	0.1257
2.2418	3950	0.1277
2.2701	4000	0.1372
2.2985	4050	0.1329
2.3269	4100	0.1347
2.3553	4150	0.1332
2.3837	4200	0.1322
2.4120	4250	0.1208
2.4404	4300	0.1311
2.4688	4350	0.1248
2.4972	4400	0.1246
2.5255	4450	0.1155
2.5539	4500	0.1243
2.5823	4550	0.1160
2.6107	4600	0.1143
2.6390	4650	0.1275
2.6674	4700	0.1258
2.6958	4750	0.1196
2.7242	4800	0.1068
2.7526	4850	0.1167
2.7809	4900	0.1181
2.8093	4950	0.1057
2.8377	5000	0.1169
2.8661	5050	0.1287
2.8944	5100	0.1108
2.9228	5150	0.1110
2.9512	5200	0.1145
2.9796	5250	0.1161
3.0079	5300	0.1172
3.0363	5350	0.1005
3.0647	5400	0.0977
3.0931	5450	0.1045
3.1215	5500	0.1021
3.1498	5550	0.1059
3.1782	5600	0.1026
3.2066	5650	0.0998
3.2350	5700	0.0982
3.2633	5750	0.1003
3.2917	5800	0.1015
3.3201	5850	0.0966
3.3485	5900	0.0971
3.3768	5950	0.1033
3.4052	6000	0.1001
3.4336	6050	0.0942
3.4620	6100	0.1028
3.4904	6150	0.0934
3.5187	6200	0.0918
3.5471	6250	0.0993
3.5755	6300	0.0943
3.6039	6350	0.1046
3.6322	6400	0.0941
3.6606	6450	0.0999
3.6890	6500	0.0998
3.7174	6550	0.0987
3.7457	6600	0.1078
3.7741	6650	0.0972
3.8025	6700	0.1008
3.8309	6750	0.1070
3.8593	6800	0.0878
3.8876	6850	0.0966
3.9160	6900	0.0909
3.9444	6950	0.0991
3.9728	7000	0.0969

Training Time

• Training: 2.5 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.14.0
• Datasets: 5.0.1
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}