AryehRotberg/ToS-Sentence-Transformers-V3

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AryehRotberg/ToS-Sentence-Transformers-V3")
# Run inference
sentences = [
    'You agree that regardless of any statute or law to the contrary, any claim or cause of action arising out of or related to use of the Desmos Services or these Terms must be filed within one (1) year after such claim or cause of action arose or be forever barred.',
    'You have a reduced time period to take legal action against the service',
    'The data retention period is kept to the minimum necessary for fulfilling its purposes',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: all-nli-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9991

Training Details

Training Dataset

Unnamed Dataset

• Size: 167,508 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 3 tokens mean: 48.65 tokens max: 256 tokens	min: 6 tokens mean: 14.89 tokens max: 29 tokens	min: 6 tokens mean: 14.24 tokens max: 29 tokens

• Samples:

  anchor	positive	negative
  This websites also uses Google Analytics, a web analysis service provided by Google Inc. ("Google"). Google Inc. is an enterprise of the holding company Alphabet Inc., domiciled in the USA	Third-party cookies are used for statistics	The service is open-source
  Terms of Use This Agreement was last revised on Dec 6, 2017.	There is a date of the last update of the agreements	Many third parties are involved in operating the service
  We reserve the right, at Our sole discretion, to modify or replace these Terms at any time. If a revision is material We will make reasonable efforts to provide at least 30 days' notice prior to any new terms taking effect.	When the service wants to make a material change to its terms, you are notified at least 30 days in advance	User-generated content is encrypted, and this service cannot decrypt it

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 41,877 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 4 tokens mean: 47.12 tokens max: 256 tokens	min: 6 tokens mean: 14.92 tokens max: 29 tokens	min: 6 tokens mean: 14.56 tokens max: 29 tokens

• Samples:

  anchor	positive	negative
  © access or search or attempt to access or search the Services by any means (automated or otherwise) other than through our currently available, published interfaces that are provided by Podyssey (and only pursuant to those terms and conditions) or unless permitted by Podyssey’s robots.txt file or other robot exclusion mechanisms. (d) scrape the Services, and particularly scrape Content (as defined below) from the Services.	Spidering, crawling, or accessing the site through any automated means is not allowed	User-generated content is encrypted, and this service cannot decrypt it
  License by Customer to Use Feedback. Customer grants to SFDC and its Affiliates a worldwide, perpetual, irrevocable, royalty-free license to use and incorporate into its services any suggestion, enhancement request, recommendation, correction or other feedback provided by Customer or Users relating to the operation of SFDC’s or its Affiliates’ services.	If you offer suggestions to the service, they may use that without your approval or compensation, but they do not become the owner	You can opt out of providing personal information to third parties
  OVPN does not log any activity when connected to our VPN service.	Only necessary logs are kept by the service to ensure quality	You agree to defend, indemnify, and hold the service harmless in case of a claim related to your use of the service

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	all-nli-dev_cosine_accuracy
-1	-1	-	-	0.9478
0.0096	100	1.34	1.1442	0.9598
0.0191	200	1.1161	0.9002	0.9725
0.0287	300	0.8731	0.7618	0.9786
0.0382	400	0.739	0.6587	0.9835
0.0478	500	0.6753	0.5901	0.9860
0.0573	600	0.6199	0.5277	0.9877
0.0669	700	0.5434	0.4952	0.9890
0.0764	800	0.4781	0.4602	0.9901
0.0860	900	0.4852	0.4351	0.9905
0.0955	1000	0.4329	0.4114	0.9910
0.1051	1100	0.4432	0.3804	0.9919
0.1146	1200	0.4224	0.3649	0.9928
0.1242	1300	0.3697	0.3488	0.9930
0.1337	1400	0.3724	0.3338	0.9936
0.1433	1500	0.3467	0.3246	0.9938
0.1528	1600	0.3728	0.3045	0.9945
0.1624	1700	0.3281	0.2952	0.9943
0.1719	1800	0.3187	0.2907	0.9946
0.1815	1900	0.336	0.2707	0.9952
0.1910	2000	0.2957	0.2667	0.9952
0.2006	2100	0.2787	0.2650	0.9955
0.2101	2200	0.2698	0.2534	0.9954
0.2197	2300	0.2741	0.2562	0.9956
0.2292	2400	0.2736	0.2477	0.9957
0.2388	2500	0.2936	0.2400	0.9960
0.2483	2600	0.2513	0.2321	0.9962
0.2579	2700	0.2564	0.2301	0.9965
0.2674	2800	0.245	0.2277	0.9965
0.2770	2900	0.2406	0.2156	0.9967
0.2865	3000	0.2074	0.2125	0.9966
0.2961	3100	0.2544	0.2081	0.9965
0.3056	3200	0.2333	0.2034	0.9968
0.3152	3300	0.2311	0.1998	0.9971
0.3247	3400	0.2294	0.1931	0.9972
0.3343	3500	0.2289	0.1877	0.9973
0.3438	3600	0.2291	0.1843	0.9974
0.3534	3700	0.2406	0.1748	0.9977
0.3629	3800	0.1851	0.1754	0.9974
0.3725	3900	0.2172	0.1691	0.9976
0.3820	4000	0.1885	0.1677	0.9979
0.3916	4100	0.2041	0.1662	0.9977
0.4011	4200	0.2052	0.1671	0.9977
0.4107	4300	0.1739	0.1626	0.9980
0.4202	4400	0.1721	0.1598	0.9979
0.4298	4500	0.1682	0.1575	0.9980
0.4394	4600	0.2076	0.1518	0.9980
0.4489	4700	0.1657	0.1549	0.9978
0.4585	4800	0.1827	0.1456	0.9981
0.4680	4900	0.1577	0.1412	0.9984
0.4776	5000	0.1869	0.1400	0.9983
0.4871	5100	0.1437	0.1400	0.9983
0.4967	5200	0.1806	0.1372	0.9982
0.5062	5300	0.1457	0.1358	0.9982
0.5158	5400	0.1529	0.1339	0.9983
0.5253	5500	0.1732	0.1300	0.9982
0.5349	5600	0.1563	0.1270	0.9984
0.5444	5700	0.1411	0.1267	0.9985
0.5540	5800	0.149	0.1270	0.9985
0.5635	5900	0.1492	0.1264	0.9985
0.5731	6000	0.1466	0.1200	0.9986
0.5826	6100	0.1423	0.1190	0.9986
0.5922	6200	0.1389	0.1204	0.9985
0.6017	6300	0.1287	0.1153	0.9984
0.6113	6400	0.1307	0.1139	0.9986
0.6208	6500	0.1383	0.1129	0.9987
0.6304	6600	0.1332	0.1105	0.9987
0.6399	6700	0.1228	0.1090	0.9988
0.6495	6800	0.119	0.1093	0.9987
0.6590	6900	0.1459	0.1076	0.9987
0.6686	7000	0.1162	0.1058	0.9988
0.6781	7100	0.1105	0.1054	0.9988
0.6877	7200	0.1379	0.1044	0.9988
0.6972	7300	0.1555	0.1017	0.9989
0.7068	7400	0.1471	0.0982	0.9989
0.7163	7500	0.1308	0.0983	0.9988
0.7259	7600	0.1095	0.0965	0.9988
0.7354	7700	0.1321	0.0956	0.9989
0.7450	7800	0.1108	0.0938	0.9987
0.7545	7900	0.1151	0.0918	0.9989
0.7641	8000	0.1179	0.0920	0.9990
0.7736	8100	0.117	0.0910	0.9991
0.7832	8200	0.1426	0.0895	0.9989
0.7927	8300	0.122	0.0891	0.9990
0.8023	8400	0.1136	0.0888	0.9989
0.8118	8500	0.0935	0.0882	0.9989
0.8214	8600	0.1143	0.0872	0.9989
0.8309	8700	0.0982	0.0873	0.9989
0.8405	8800	0.1171	0.0857	0.9989
0.8500	8900	0.1091	0.0844	0.9989
0.8596	9000	0.1046	0.0840	0.9989
0.8691	9100	0.0897	0.0836	0.9990
0.8787	9200	0.0804	0.0832	0.9991
0.8883	9300	0.0967	0.0827	0.9991
0.8978	9400	0.0897	0.0820	0.9991
0.9074	9500	0.0968	0.0813	0.9990
0.9169	9600	0.1108	0.0814	0.9991
0.9265	9700	0.1058	0.0806	0.9991
0.9360	9800	0.0871	0.0800	0.9990
0.9456	9900	0.1079	0.0797	0.9991
0.9551	10000	0.1064	0.0794	0.9991
0.9647	10100	0.1095	0.0792	0.9991
0.9742	10200	0.0858	0.0791	0.9991
0.9838	10300	0.0997	0.0791	0.9991
0.9933	10400	0.0888	0.0791	0.9991

Framework Versions

• Python: 3.12.7
• Sentence Transformers: 4.1.0
• Transformers: 4.51.3
• PyTorch: 2.4.1+cu124
• Accelerate: 1.6.0
• Datasets: 3.5.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}