--- tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:1000000 - loss:MultipleNegativesRankingLoss base_model: sentence-transformers/LaBSE widget: - source_sentence: Акӑ ӗнтӗ Чакак кимӗ ҫине сикрӗ, Коля пӗр-икӗ хут шнуртан туртрӗ те, мотор кӗрлесе те кайрӗ, унтан кимӗ утрав еннелле вӗҫтерчӗ. sentences: - Вот Сорока вскочил в лодку, Коля дернул за шнур, раз, другой, мотор затрещал, и лодка понеслась к острову. - Победа римского флота в гавани Эвносте. - Повесть Бориса Горбатова о подвиге и героизме советских людей во время Великой Отечественной войны. - source_sentence: Ун патне пысӑках мар хырӑмлӑ, шурӑ сӑнлӑ, хӗрлӗ питлӗ, лутра ҫын килсе кӗчӗ. sentences: - Антонов, Семён Михеевич - Явился низенький человек, с умеренным брюшком, с белым лицом, румяными щеками - Чёрно-белые фильмы СССР - source_sentence: '3. Анчах Гаваон ҫыннисем, Иисус Иерихонпа Гай хулисене епле пӗтерсе тӑкни ҫинчен илтсессӗн, 4. акӑ мӗнле чеелӗх тупнӑ: ашакӗсем ҫине ҫул валли кивӗ михӗсемпе ҫӑкӑр янтӑласа хунӑ, ҫӗтӗлсе пӗтнӗ, саплӑклӑ тир хутаҫпа эрех илнӗ; 5. ури сырри те вӗсен кивӗ, саплӑклӑ пулнӑ, ҫийӗнчи тумтирӗсем те ҫӗтӗк пулнӑ; ҫул ҫине илнӗ ҫӑкӑрӗ те пӗтӗмпех типсе-кӑвакарса кайнӑскер, [тӗпренсе] пӗтнӗскер пулнӑ.' sentences: - '3. Но жители Гаваона, услышав, что Иисус сделал с Иерихоном и Гаем, 4. употребили хитрость: пошли, запаслись хлебом на дорогу и положили ветхие мешки на ослов своих и ветхие, изорванные и заплатанные мехи вина; 5. и обувь на ногах их была ветхая с заплатами, и одежда на них ветхая; и весь дорожный хлеб их был сухой и заплесневелый [и раскрошенный].' - «Черти бы их дули!..» — в отчаянии вскричал Щукарь и кинулся к цыганскому табору, но, выскочив на пригорок, обнаружил, что ни шатров, ни кибиток возле речки уже нет. - 9. И сделаю над тобою то, чего Я никогда не делал и чему подобного впредь не буду делать, за все твои мерзости. - source_sentence: Эпӗ кӗпер айӗпе укҫасӑрах, ахалех вӗҫсе тухрӑм. sentences: - У меня в экипаже был механик — что называется, «палец в рот не клади». - А я под мост даром слетал. - Я пользовался этим и прогуливал школу, чтобы проводить время в компании более старших ребят. - source_sentence: Генри Джастис Форд sentences: - — Вижу, по одному делу? — спросила она, взглянув на Сашу и его приятелей. - Я вышел из ванны свеж и бодр, как будто собирался на бал. - Форд, Генри Джастис pipeline_tag: sentence-similarity library_name: sentence-transformers --- # SentenceTransformer based on sentence-transformers/LaBSE This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE). It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [sentence-transformers/LaBSE](https://huggingface.co/sentence-transformers/LaBSE) - **Maximum Sequence Length:** 256 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Dense({'in_features': 768, 'out_features': 768, 'bias': True, 'activation_function': 'torch.nn.modules.activation.Tanh'}) (3): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("sentence_transformers_model_id") # Run inference sentences = [ 'Генри Джастис Форд', 'Форд, Генри Джастис', 'Я вышел из ванны свеж и бодр, как будто собирался на бал.', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 1,000,000 training samples * Columns: sentence_0, sentence_1, and label * Approximate statistics based on the first 1000 samples: | | sentence_0 | sentence_1 | label | |:--------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:--------------------------------------------------------------| | type | string | string | float | | details | | | | * Samples: | sentence_0 | sentence_1 | label | |:-----------------------------------------------------------------------------------|:--------------------------------------------------------------|:-----------------| | Темех мар. | Дело десятое. | 1.0 | | Уругвайӑн тĕн ĕҫченĕсем | Религиозные деятели Уругвая | 1.0 | | Эп аванах ас тӑватӑп, пилӗк ҫул каялла пахчана эпир лайӑх тасатнӑччӗ. | А пять лет тому назад я знал, что сад был чищен. | 1.0 | * Loss: [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim" } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 12 - `per_device_eval_batch_size`: 12 - `num_train_epochs`: 1 - `fp16`: True - `multi_dataset_batch_sampler`: round_robin #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 12 - `per_device_eval_batch_size`: 12 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 5e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.0 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: True - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `tp_size`: 0 - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: round_robin
### Training Logs
Click to expand | Epoch | Step | Training Loss | |:------:|:-----:|:-------------:| | 0.0012 | 100 | - | | 0.0024 | 200 | - | | 0.0036 | 300 | - | | 0.0048 | 400 | - | | 0.0060 | 500 | 0.5331 | | 0.0072 | 600 | - | | 0.0084 | 700 | - | | 0.0096 | 800 | - | | 0.0108 | 900 | - | | 0.0120 | 1000 | 0.3694 | | 0.0132 | 1100 | - | | 0.0144 | 1200 | - | | 0.0156 | 1300 | - | | 0.0168 | 1400 | - | | 0.0180 | 1500 | 0.3141 | | 0.0192 | 1600 | - | | 0.0204 | 1700 | - | | 0.0216 | 1800 | - | | 0.0228 | 1900 | - | | 0.0240 | 2000 | 0.2836 | | 0.0252 | 2100 | - | | 0.0264 | 2200 | - | | 0.0276 | 2300 | - | | 0.0288 | 2400 | - | | 0.0300 | 2500 | 0.2823 | | 0.0312 | 2600 | - | | 0.0324 | 2700 | - | | 0.0336 | 2800 | - | | 0.0348 | 2900 | - | | 0.0360 | 3000 | 0.265 | | 0.0372 | 3100 | - | | 0.0384 | 3200 | - | | 0.0396 | 3300 | - | | 0.0408 | 3400 | - | | 0.0420 | 3500 | 0.2599 | | 0.0432 | 3600 | - | | 0.0444 | 3700 | - | | 0.0456 | 3800 | - | | 0.0468 | 3900 | - | | 0.0480 | 4000 | 0.234 | | 0.0492 | 4100 | - | | 0.0504 | 4200 | - | | 0.0516 | 4300 | - | | 0.0528 | 4400 | - | | 0.0540 | 4500 | 0.1966 | | 0.0552 | 4600 | - | | 0.0564 | 4700 | - | | 0.0576 | 4800 | - | | 0.0588 | 4900 | - | | 0.0600 | 5000 | 0.2204 | | 0.0612 | 5100 | - | | 0.0624 | 5200 | - | | 0.0636 | 5300 | - | | 0.0648 | 5400 | - | | 0.0660 | 5500 | 0.2272 | | 0.0672 | 5600 | - | | 0.0684 | 5700 | - | | 0.0696 | 5800 | - | | 0.0708 | 5900 | - | | 0.0720 | 6000 | 0.2256 | | 0.0732 | 6100 | - | | 0.0744 | 6200 | - | | 0.0756 | 6300 | - | | 0.0768 | 6400 | - | | 0.0780 | 6500 | 0.2071 | | 0.0792 | 6600 | - | | 0.0804 | 6700 | - | | 0.0816 | 6800 | - | | 0.0828 | 6900 | - | | 0.0840 | 7000 | 0.2113 | | 0.0852 | 7100 | - | | 0.0864 | 7200 | - | | 0.0876 | 7300 | - | | 0.0888 | 7400 | - | | 0.0900 | 7500 | 0.2222 | | 0.0912 | 7600 | - | | 0.0924 | 7700 | - | | 0.0936 | 7800 | - | | 0.0948 | 7900 | - | | 0.0960 | 8000 | 0.2186 | | 0.0972 | 8100 | - | | 0.0984 | 8200 | - | | 0.0996 | 8300 | - | | 0.1008 | 8400 | - | | 0.1020 | 8500 | 0.2137 | | 0.1032 | 8600 | - | | 0.1044 | 8700 | - | | 0.1056 | 8800 | - | | 0.1068 | 8900 | - | | 0.1080 | 9000 | 0.1928 | | 0.1092 | 9100 | - | | 0.1104 | 9200 | - | | 0.1116 | 9300 | - | | 0.1128 | 9400 | - | | 0.1140 | 9500 | 0.2117 | | 0.1152 | 9600 | - | | 0.1164 | 9700 | - | | 0.1176 | 9800 | - | | 0.1188 | 9900 | - | | 0.1200 | 10000 | 0.1987 | | 0.1212 | 10100 | - | | 0.1224 | 10200 | - | | 0.1236 | 10300 | - | | 0.1248 | 10400 | - | | 0.1260 | 10500 | 0.2011 | | 0.1272 | 10600 | - | | 0.1284 | 10700 | - | | 0.1296 | 10800 | - | | 0.1308 | 10900 | - | | 0.1320 | 11000 | 0.1775 | | 0.1332 | 11100 | - | | 0.1344 | 11200 | - | | 0.1356 | 11300 | - | | 0.1368 | 11400 | - | | 0.1380 | 11500 | 0.2048 | | 0.1392 | 11600 | - | | 0.1404 | 11700 | - | | 0.1416 | 11800 | - | | 0.1428 | 11900 | - | | 0.1440 | 12000 | 0.2064 | | 0.1452 | 12100 | - | | 0.1464 | 12200 | - | | 0.1476 | 12300 | - | | 0.1488 | 12400 | - | | 0.1500 | 12500 | 0.1883 | | 0.1512 | 12600 | - | | 0.1524 | 12700 | - | | 0.1536 | 12800 | - | | 0.1548 | 12900 | - | | 0.1560 | 13000 | 0.2084 | | 0.1572 | 13100 | - | | 0.1584 | 13200 | - | | 0.1596 | 13300 | - | | 0.1608 | 13400 | - | | 0.1620 | 13500 | 0.2077 | | 0.1632 | 13600 | - | | 0.1644 | 13700 | - | | 0.1656 | 13800 | - | | 0.1668 | 13900 | - | | 0.1680 | 14000 | 0.1866 | | 0.1692 | 14100 | - | | 0.1704 | 14200 | - | | 0.1716 | 14300 | - | | 0.1728 | 14400 | - | | 0.1740 | 14500 | 0.1859 | | 0.1752 | 14600 | - | | 0.1764 | 14700 | - | | 0.1776 | 14800 | - | | 0.1788 | 14900 | - | | 0.1800 | 15000 | 0.1735 | | 0.1812 | 15100 | - | | 0.1824 | 15200 | - | | 0.1836 | 15300 | - | | 0.1848 | 15400 | - | | 0.1860 | 15500 | 0.171 | | 0.1872 | 15600 | - | | 0.1884 | 15700 | - | | 0.1896 | 15800 | - | | 0.1908 | 15900 | - | | 0.1920 | 16000 | 0.1465 | | 0.1932 | 16100 | - | | 0.1944 | 16200 | - | | 0.1956 | 16300 | - | | 0.1968 | 16400 | - | | 0.1980 | 16500 | 0.1921 | | 0.1992 | 16600 | - | | 0.2004 | 16700 | - | | 0.2016 | 16800 | - | | 0.2028 | 16900 | - | | 0.2040 | 17000 | 0.1669 | | 0.2052 | 17100 | - | | 0.2064 | 17200 | - | | 0.2076 | 17300 | - | | 0.2088 | 17400 | - | | 0.2100 | 17500 | 0.1656 | | 0.2112 | 17600 | - | | 0.2124 | 17700 | - | | 0.2136 | 17800 | - | | 0.2148 | 17900 | - | | 0.2160 | 18000 | 0.1952 | | 0.2172 | 18100 | - | | 0.2184 | 18200 | - | | 0.2196 | 18300 | - | | 0.2208 | 18400 | - | | 0.2220 | 18500 | 0.1658 | | 0.2232 | 18600 | - | | 0.2244 | 18700 | - | | 0.2256 | 18800 | - | | 0.2268 | 18900 | - | | 0.2280 | 19000 | 0.1774 | | 0.2292 | 19100 | - | | 0.2304 | 19200 | - | | 0.2316 | 19300 | - | | 0.2328 | 19400 | - | | 0.2340 | 19500 | 0.1802 | | 0.2352 | 19600 | - | | 0.2364 | 19700 | - | | 0.2376 | 19800 | - | | 0.2388 | 19900 | - | | 0.2400 | 20000 | 0.1724 | | 0.2412 | 20100 | - | | 0.2424 | 20200 | - | | 0.2436 | 20300 | - | | 0.2448 | 20400 | - | | 0.2460 | 20500 | 0.1653 | | 0.2472 | 20600 | - | | 0.2484 | 20700 | - | | 0.2496 | 20800 | - | | 0.2508 | 20900 | - | | 0.2520 | 21000 | 0.1484 | | 0.2532 | 21100 | - | | 0.2544 | 21200 | - | | 0.2556 | 21300 | - | | 0.2568 | 21400 | - | | 0.2580 | 21500 | 0.1544 | | 0.2592 | 21600 | - | | 0.2604 | 21700 | - | | 0.2616 | 21800 | - | | 0.2628 | 21900 | - | | 0.2640 | 22000 | 0.174 | | 0.2652 | 22100 | - | | 0.2664 | 22200 | - | | 0.2676 | 22300 | - | | 0.2688 | 22400 | - | | 0.2700 | 22500 | 0.1488 | | 0.2712 | 22600 | - | | 0.2724 | 22700 | - | | 0.2736 | 22800 | - | | 0.2748 | 22900 | - | | 0.2760 | 23000 | 0.1696 | | 0.2772 | 23100 | - | | 0.2784 | 23200 | - | | 0.2796 | 23300 | - | | 0.2808 | 23400 | - | | 0.2820 | 23500 | 0.1468 | | 0.2832 | 23600 | - | | 0.2844 | 23700 | - | | 0.2856 | 23800 | - | | 0.2868 | 23900 | - | | 0.2880 | 24000 | 0.1738 | | 0.2892 | 24100 | - | | 0.2904 | 24200 | - | | 0.2916 | 24300 | - | | 0.2928 | 24400 | - | | 0.2940 | 24500 | 0.1667 | | 0.2952 | 24600 | - | | 0.2964 | 24700 | - | | 0.2976 | 24800 | - | | 0.2988 | 24900 | - | | 0.3000 | 25000 | 0.1562 | | 0.3012 | 25100 | - | | 0.3024 | 25200 | - | | 0.3036 | 25300 | - | | 0.3048 | 25400 | - | | 0.3060 | 25500 | 0.1628 | | 0.3072 | 25600 | - | | 0.3084 | 25700 | - | | 0.3096 | 25800 | - | | 0.3108 | 25900 | - | | 0.3120 | 26000 | 0.1392 | | 0.3132 | 26100 | - | | 0.3144 | 26200 | - |
### Framework Versions - Python: 3.12.10 - Sentence Transformers: 4.1.0 - Transformers: 4.51.3 - PyTorch: 2.6.0+cu124 - Accelerate: 1.8.1 - Datasets: 3.6.0 - Tokenizers: 0.21.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```