ayushexel/embed-all-MiniLM-L6-v2-squad-8-epochs

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ayushexel/embed-all-MiniLM-L6-v2-squad-8-epochs")
# Run inference
sentences = [
    'When was the boundaries of the city last adjusted? ',
    'In 1854 president Antonio López de Santa Anna enlarged the area of the Federal District almost eightfold from the original 220 to 1,700 km2 (80 to 660 sq mi), annexing the rural and mountainous areas to secure the strategic mountain passes to the south and southwest to protect the city in event of a foreign invasion. (The Mexican–American War had just been fought.) The last changes to the limits of the Federal District were made between 1898 and 1902, reducing the area to the current 1,479 km2 (571 sq mi) by adjusting the southern border with the state of Morelos. By that time, the total number of municipalities within the Federal District was twenty-two.',
    'The Mayor and council members are elected to four-year terms. The City Council is a unicameral body consisting of 51 council members whose districts are defined by geographic population boundaries. Each term for the mayor and council members lasts four years and has a three consecutive-term limit, but can resume after a four-year break. The New York City Administrative Code, the New York City Rules, and the City Record are the code of local laws, compilation of regulations, and official journal, respectively.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: gooqa-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.4028

Training Details

Training Dataset

Unnamed Dataset

• Size: 44,284 training samples
• Columns: question, context, and negative
• Approximate statistics based on the first 1000 samples:

  	question	context	negative
  type	string	string	string
  details	min: 7 tokens mean: 14.52 tokens max: 40 tokens	min: 29 tokens mean: 148.3 tokens max: 256 tokens	min: 28 tokens mean: 151.46 tokens max: 256 tokens

• Samples:

  question	context	negative
  Which country needs a similarity requirement to determine if the work was copied?	In the U.S., for example, copyright case law contains a substantial similarity requirement to determine whether the work was copied. Likewise, courts may require computer software to pass an Abstraction-Filtration-Comparison test (AFC Test) to determine if it is too abstract to qualify for protection, or too dissimilar to an original work to be considered infringing. Software-related case law has also clarified that the amount of R&D, effort and expense put into a work's creation doesn't affect copyright protection.	Outsourcing production to low wage countries like Bangladesh, China, India and Sri Lanka became possible when the Multi Fibre Agreement (MFA) was abolished. The MFA, which placed quotas on textiles imports, was deemed a protectionist measure.[citation needed] Globalization is often quoted as the single most contributing factor to the poor working conditions of garment workers. Although many countries recognize treaties like the International Labor Organization, which attempt to set standards for worker safety and rights, many countries have made exceptions to certain parts of the treaties or failed to thoroughly enforce them. India for example has not ratified sections 87 and 92 of the treaty.[citation needed]
  What other OS could use Internet Explorer?	Internet Explorer, on the other hand, was bundled free with the Windows operating system (and was also downloadable free), and therefore it was funded partly by the sales of Windows to computer manufacturers and direct to users. Internet Explorer also used to be available for the Mac. It is likely that releasing IE for the Mac was part of Microsoft's overall strategy to fight threats to its quasi-monopoly platform dominance - threats such as web standards and Java - by making some web developers, or at least their managers, assume that there was "no need" to develop for anything other than Internet Explorer. In this respect, IE may have contributed to Windows and Microsoft applications sales in another way, through "lock-in" to Microsoft's browser.	The most recent major entrant to the browser market is Chrome, first released in September 2008. Chrome's take-up has increased significantly year by year, by doubling its usage share from 8% to 16% by August 2011. This increase seems largely to be at the expense of Internet Explorer, whose share has tended to decrease from month to month. In December 2011, Chrome overtook Internet Explorer 8 as the most widely used web browser but still had lower usage than all versions of Internet Explorer combined. Chrome's user-base continued to grow and in May 2012, Chrome's usage passed the usage of all versions of Internet Explorer combined. By April 2014, Chrome's usage had hit 45%.
  When was the first British Mont Blanc ascent?	The first British Mont Blanc ascent was in 1788; the first female ascent in 1819. By the mid-1850s Swiss mountaineers had ascended most of the peaks and were eagerly sought as mountain guides. Edward Whymper reached the top of the Matterhorn in 1865 (after seven attempts), and in 1938 the last of the six great north faces of the Alps was climbed with the first ascent of the Eiger Nordwand (north face of the Eiger).	While Mont Blanc was first climbed in 1786, most of the Alpine four-thousanders were climbed during the first half of the 19th century; the ascent of the Matterhorn in 1865 marked the end of the golden age of alpinism. Karl Blodig (1859–1956) was among the first to successfully climb all the major 4,000 m peaks. He completed his series of ascents in 1911.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 5,000 evaluation samples
• Columns: question, context, and negative_1
• Approximate statistics based on the first 1000 samples:

  	question	context	negative_1
  type	string	string	string
  details	min: 7 tokens mean: 14.35 tokens max: 38 tokens	min: 32 tokens mean: 147.73 tokens max: 256 tokens	min: 32 tokens mean: 144.23 tokens max: 256 tokens

• Samples:

  question	context	negative_1
  What industry having to do with shopping benefits from DST?	The practice has received both advocacy and criticism. Putting clocks forward benefits retailing, sports, and other activities that exploit sunlight after working hours, but can cause problems for evening entertainment and for other activities tied to sunlight, such as farming. Although some early proponents of DST aimed to reduce evening use of incandescent lighting, which used to be a primary use of electricity, modern heating and cooling usage patterns differ greatly and research about how DST affects energy use is limited or contradictory.	Opponents argue that actual energy savings are inconclusive, that DST increases health risks such as heart attack, that DST can disrupt morning activities, and that the act of changing clocks twice a year is economically and socially disruptive and cancels out any benefit. Farmers have tended to oppose DST.
  Boston was chartered as what in 1822?	In 1822, the citizens of Boston voted to change the official name from "the Town of Boston" to "the City of Boston", and on March 4, 1822, the people of Boston accepted the charter incorporating the City. At the time Boston was chartered as a city, the population was about 46,226, while the area of the city was only 4.7 square miles (12 km2).	In 1822, the citizens of Boston voted to change the official name from "the Town of Boston" to "the City of Boston", and on March 4, 1822, the people of Boston accepted the charter incorporating the City. At the time Boston was chartered as a city, the population was about 46,226, while the area of the city was only 4.7 square miles (12 km2).
  What caused Steve Jobs to resign from Apple in 1985?	Burrel's innovative design, which combined the low production cost of an Apple II with the computing power of Lisa's CPU, the Motorola 68K, received the attention of Steve Jobs, co-founder of Apple. Realizing that the Macintosh was more marketable than the Lisa, he began to focus his attention on the project. Raskin left the team in 1981 over a personality conflict with Jobs. Team member Andy Hertzfeld said that the final Macintosh design is closer to Jobs' ideas than Raskin's. After hearing of the pioneering GUI technology being developed at Xerox PARC, Jobs had negotiated a visit to see the Xerox Alto computer and its Smalltalk development tools in exchange for Apple stock options. The Lisa and Macintosh user interfaces were influenced by technology seen at Xerox PARC and were combined with the Macintosh group's own ideas. Jobs also commissioned industrial designer Hartmut Esslinger to work on the Macintosh line, resulting in the "Snow White" design language; although it came too lat...	Burrel's innovative design, which combined the low production cost of an Apple II with the computing power of Lisa's CPU, the Motorola 68K, received the attention of Steve Jobs, co-founder of Apple. Realizing that the Macintosh was more marketable than the Lisa, he began to focus his attention on the project. Raskin left the team in 1981 over a personality conflict with Jobs. Team member Andy Hertzfeld said that the final Macintosh design is closer to Jobs' ideas than Raskin's. After hearing of the pioneering GUI technology being developed at Xerox PARC, Jobs had negotiated a visit to see the Xerox Alto computer and its Smalltalk development tools in exchange for Apple stock options. The Lisa and Macintosh user interfaces were influenced by technology seen at Xerox PARC and were combined with the Macintosh group's own ideas. Jobs also commissioned industrial designer Hartmut Esslinger to work on the Macintosh line, resulting in the "Snow White" design language; although it came too lat...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• num_train_epochs: 8
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 8
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooqa-dev_cosine_accuracy
-1	-1	-	-	0.3296
0.2890	100	0.4373	0.8053	0.3796
0.5780	200	0.3961	0.7807	0.3898
0.8671	300	0.3941	0.7710	0.3892
1.1561	400	0.3341	0.7540	0.4056
1.4451	500	0.3039	0.7541	0.3972
1.7341	600	0.2992	0.7479	0.4098
2.0231	700	0.2822	0.7496	0.4062
2.3121	800	0.2081	0.7517	0.4032
2.6012	900	0.2128	0.7461	0.3984
2.8902	1000	0.213	0.7457	0.4034
3.1792	1100	0.1814	0.7524	0.4096
3.4682	1200	0.1601	0.7517	0.4078
3.7572	1300	0.1662	0.7470	0.4096
4.0462	1400	0.1631	0.7528	0.4046
4.3353	1500	0.1365	0.7565	0.4060
4.6243	1600	0.1301	0.7563	0.4044
4.9133	1700	0.1375	0.7593	0.4020
5.2023	1800	0.1201	0.7663	0.4088
5.4913	1900	0.114	0.7592	0.4072
5.7803	2000	0.1186	0.7609	0.4086
6.0694	2100	0.1129	0.7583	0.4098
6.3584	2200	0.1041	0.7594	0.4030
6.6474	2300	0.1029	0.7603	0.4070
6.9364	2400	0.1036	0.7622	0.4048
7.2254	2500	0.0962	0.7651	0.4104
7.5145	2600	0.0977	0.7630	0.4060
7.8035	2700	0.0956	0.7627	0.4090
-1	-1	-	-	0.4028

Framework Versions

• Python: 3.11.0
• Sentence Transformers: 4.0.1
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}