all-MiniLM-L6-v11-pair_score

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'semi polished candle holder',
    'long lasting candle holder',
    'ice cream',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 2
• warmup_ratio: 0.1
• fp16: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss
1.4755	100600	3.8748
1.4770	100700	4.053
1.4784	100800	3.9575
1.4799	100900	4.1901
1.4814	101000	4.0269
1.4828	101100	3.7069
1.4843	101200	4.1148
1.4858	101300	3.9824
1.4872	101400	3.7988
1.4887	101500	3.7947
1.4902	101600	4.0534
1.4916	101700	3.7761
1.4931	101800	4.237
1.4946	101900	4.4075
1.4960	102000	3.9308
1.4975	102100	3.7867
1.4990	102200	4.3526
1.5004	102300	4.1362
1.5019	102400	3.997
1.5034	102500	4.3269
1.5048	102600	3.9814
1.5063	102700	3.8537
1.5078	102800	3.9149
1.5092	102900	3.7611
1.5107	103000	3.7806
1.5122	103100	4.2381
1.5136	103200	4.0435
1.5151	103300	3.6451
1.5166	103400	3.8354
1.5180	103500	3.9206
1.5195	103600	3.9973
1.5210	103700	3.9566
1.5224	103800	4.0192
1.5239	103900	3.7938
1.5254	104000	4.2643
1.5268	104100	4.1637
1.5283	104200	4.0284
1.5298	104300	4.21
1.5312	104400	3.4641
1.5327	104500	3.6869
1.5342	104600	3.8448
1.5356	104700	4.2046
1.5371	104800	3.645
1.5386	104900	4.0126
1.5400	105000	3.8847
1.5415	105100	3.8365
1.5430	105200	4.185
1.5444	105300	4.1772
1.5459	105400	3.7761
1.5474	105500	3.9277
1.5488	105600	3.8262
1.5503	105700	3.9907
1.5518	105800	4.0857
1.5532	105900	4.2328
1.5547	106000	3.9554
1.5562	106100	3.8267
1.5576	106200	3.9999
1.5591	106300	4.1154
1.5606	106400	3.9688
1.5620	106500	4.0721
1.5635	106600	4.3463
1.5650	106700	3.6566
1.5664	106800	4.1804
1.5679	106900	3.7478
1.5694	107000	3.3921
1.5708	107100	3.6867
1.5723	107200	4.5457
1.5738	107300	3.6444
1.5752	107400	3.7478
1.5767	107500	3.7962
1.5782	107600	4.13
1.5796	107700	3.7226
1.5811	107800	3.7272
1.5826	107900	3.5184
1.5840	108000	4.0702
1.5855	108100	4.4565
1.5870	108200	3.6692
1.5884	108300	4.0094
1.5899	108400	3.7197
1.5914	108500	3.7295
1.5928	108600	3.5424
1.5943	108700	4.0009
1.5958	108800	3.9083
1.5972	108900	4.0579
1.5987	109000	3.8253
1.6002	109100	3.8134
1.6016	109200	3.9665
1.6031	109300	3.8888
1.6046	109400	3.966
1.6060	109500	4.1187
1.6075	109600	3.9186
1.6090	109700	3.6485
1.6104	109800	3.8329
1.6119	109900	3.824
1.6134	110000	3.501
1.6148	110100	3.8698
1.6163	110200	4.0928
1.6178	110300	3.7599
1.6192	110400	3.8688
1.6207	110500	3.5656
1.6222	110600	4.1954
1.6236	110700	3.9274
1.6251	110800	3.9158
1.6266	110900	3.7125
1.6280	111000	4.0304
1.6295	111100	3.5408
1.6310	111200	3.9439
1.6324	111300	3.7155
1.6339	111400	4.035
1.6354	111500	3.9391
1.6368	111600	3.8866
1.6383	111700	4.0672
1.6398	111800	4.1916
1.6412	111900	4.1134
1.6427	112000	4.3825
1.6442	112100	3.8469
1.6456	112200	3.984
1.6471	112300	4.0895
1.6486	112400	3.688
1.6500	112500	3.6982
1.6515	112600	3.6685
1.6530	112700	4.1674
1.6544	112800	4.0703
1.6559	112900	3.5716
1.6574	113000	3.9674
1.6588	113100	4.1678
1.6603	113200	3.9769
1.6618	113300	3.8312
1.6632	113400	3.8692
1.6647	113500	3.924
1.6662	113600	4.0122
1.6676	113700	4.0432
1.6691	113800	3.8391
1.6706	113900	4.4089
1.6720	114000	3.7079
1.6735	114100	3.4194
1.6750	114200	4.2441
1.6764	114300	3.7279
1.6779	114400	3.8588
1.6794	114500	3.8865
1.6808	114600	3.6613
1.6823	114700	3.8352
1.6838	114800	4.0586
1.6852	114900	3.7488
1.6867	115000	3.7452
1.6882	115100	3.6076
1.6896	115200	3.968
1.6911	115300	4.2497
1.6926	115400	3.9571
1.6940	115500	3.6752
1.6955	115600	3.642
1.6970	115700	3.9887
1.6984	115800	3.7685
1.6999	115900	3.8536
1.7014	116000	4.081
1.7028	116100	4.192
1.7043	116200	4.081
1.7058	116300	3.8161
1.7072	116400	3.8421
1.7087	116500	3.7503
1.7102	116600	3.7952
1.7116	116700	4.1302
1.7131	116800	3.7091
1.7146	116900	4.0009
1.7160	117000	3.5709
1.7175	117100	3.954
1.7190	117200	4.0199
1.7204	117300	3.5756
1.7219	117400	3.8475
1.7234	117500	3.7051
1.7248	117600	3.5641
1.7263	117700	3.925
1.7278	117800	3.807
1.7292	117900	3.9412
1.7307	118000	3.8442
1.7322	118100	3.6595
1.7336	118200	3.8921
1.7351	118300	3.7817
1.7366	118400	3.9047
1.7380	118500	3.571
1.7395	118600	3.7622
1.7410	118700	3.8685
1.7424	118800	3.9514
1.7439	118900	4.055
1.7454	119000	3.6103
1.7468	119100	4.0892
1.7483	119200	3.7731
1.7498	119300	3.8561
1.7512	119400	4.1297
1.7527	119500	4.1099
1.7542	119600	4.1831
1.7556	119700	3.8201
1.7571	119800	3.9437
1.7586	119900	3.9154
1.7600	120000	3.9049
1.7615	120100	3.4697
1.7630	120200	3.8302
1.7644	120300	3.8078
1.7659	120400	3.7894
1.7674	120500	3.9003
1.7688	120600	3.8005
1.7703	120700	3.6398
1.7718	120800	4.0086
1.7732	120900	3.8627
1.7747	121000	3.5175
1.7762	121100	3.7655
1.7776	121200	3.8143
1.7791	121300	4.1813
1.7806	121400	3.7133
1.7820	121500	3.5701
1.7835	121600	3.3709
1.7850	121700	3.8726
1.7864	121800	3.9624
1.7879	121900	3.8097
1.7894	122000	3.9329
1.7908	122100	3.6713
1.7923	122200	4.1905
1.7938	122300	3.881
1.7952	122400	3.7906
1.7967	122500	3.8061
1.7982	122600	4.0411
1.7996	122700	3.6913
1.8011	122800	4.3033
1.8026	122900	3.84
1.8040	123000	3.8916
1.8055	123100	3.8984
1.8070	123200	3.8142
1.8084	123300	3.5308
1.8099	123400	3.7902
1.8114	123500	3.9144
1.8128	123600	4.1585
1.8143	123700	3.7845
1.8158	123800	4.0398
1.8172	123900	3.7276
1.8187	124000	3.9387
1.8202	124100	3.3395
1.8216	124200	3.8677
1.8231	124300	3.7779
1.8246	124400	3.6872
1.8260	124500	3.8913
1.8275	124600	3.7367
1.8290	124700	4.3299
1.8304	124800	3.8683
1.8319	124900	3.896
1.8334	125000	4.0298
1.8348	125100	3.6089
1.8363	125200	3.5678
1.8378	125300	4.215
1.8392	125400	3.5253
1.8407	125500	3.8163
1.8422	125600	3.8711
1.8436	125700	3.9141
1.8451	125800	4.0036
1.8466	125900	3.4214
1.8480	126000	3.7861
1.8495	126100	3.7758
1.8510	126200	3.9433
1.8524	126300	3.5923
1.8539	126400	3.8646
1.8554	126500	4.4035
1.8568	126600	3.9414
1.8583	126700	3.7132
1.8598	126800	4.3201
1.8612	126900	3.5453
1.8627	127000	3.7816
1.8642	127100	3.6934
1.8656	127200	3.8439
1.8671	127300	3.6114
1.8686	127400	4.1551
1.8700	127500	4.0338
1.8715	127600	3.9158
1.8730	127700	3.7997
1.8744	127800	3.9272
1.8759	127900	3.6009
1.8774	128000	3.8861
1.8789	128100	3.7981
1.8803	128200	3.8183
1.8818	128300	3.975
1.8833	128400	3.4799
1.8847	128500	3.7114
1.8862	128600	3.9392
1.8877	128700	3.7769
1.8891	128800	3.809
1.8906	128900	3.9282
1.8921	129000	4.2751
1.8935	129100	3.8462
1.8950	129200	3.7266
1.8965	129300	3.9677
1.8979	129400	4.0947
1.8994	129500	3.7295
1.9009	129600	3.8264
1.9023	129700	3.8546
1.9038	129800	4.0043
1.9053	129900	3.7995
1.9067	130000	3.7738
1.9082	130100	3.7783
1.9097	130200	3.8013
1.9111	130300	3.8089
1.9126	130400	3.865
1.9141	130500	3.8448
1.9155	130600	3.8157
1.9170	130700	4.0972
1.9185	130800	3.6567
1.9199	130900	3.7441
1.9214	131000	4.0964
1.9229	131100	3.5458
1.9243	131200	4.0343
1.9258	131300	3.681
1.9273	131400	3.8044
1.9287	131500	3.9433
1.9302	131600	3.6754
1.9317	131700	4.3985
1.9331	131800	3.8511
1.9346	131900	4.1095
1.9361	132000	3.2248
1.9375	132100	4.2346
1.9390	132200	3.6429
1.9405	132300	3.9987
1.9419	132400	3.9853
1.9434	132500	3.7949
1.9449	132600	3.6875
1.9463	132700	3.4602
1.9478	132800	3.7481
1.9493	132900	4.1038
1.9507	133000	3.7001
1.9522	133100	3.6621
1.9537	133200	3.5483
1.9551	133300	3.7601
1.9566	133400	3.9289
1.9581	133500	3.8237
1.9595	133600	3.6393
1.9610	133700	3.8376
1.9625	133800	3.7313
1.9639	133900	3.7408
1.9654	134000	3.7284
1.9669	134100	3.7694
1.9683	134200	3.8033
1.9698	134300	3.7479
1.9713	134400	3.6453
1.9727	134500	4.1746
1.9742	134600	3.8442
1.9757	134700	3.8437
1.9771	134800	4.0413
1.9786	134900	3.6456
1.9801	135000	3.9108
1.9815	135100	3.617
1.9830	135200	3.7945
1.9845	135300	3.8036
1.9859	135400	4.0028
1.9874	135500	3.8012
1.9889	135600	3.7975
1.9903	135700	4.0858
1.9918	135800	3.7136
1.9933	135900	3.5602
1.9947	136000	3.6315
1.9962	136100	3.7648
1.9977	136200	3.7992
1.9991	136300	3.7614

Framework Versions

• Python: 3.8.10
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.4.1+cu118
• Accelerate: 1.0.1
• Datasets: 3.0.1
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}