ayushexel/emb-all-MiniLM-L6-v2-squad-4-epochs

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ayushexel/emb-all-MiniLM-L6-v2-squad-4-epochs")
# Run inference
sentences = [
    'In the Papal States, what is the name for the specified neighborhoods where Jews were required to live?',
    'In the Papal States, which existed until 1870, Jews were required to live only in specified neighborhoods called ghettos. In the 19th and (before the end of World War II) 20th centuries, the Roman Catholic Church adhered to a distinction between "good antisemitism" and "bad antisemitism". The "bad" kind promoted hatred of Jews because of their descent. This was considered un-Christian because the Christian message was intended for all of humanity regardless of ethnicity; anyone could become a Christian. The "good" kind criticized alleged Jewish conspiracies to control newspapers, banks, and other institutions, to care only about accumulation of wealth, etc.',
    'In the Papal States, which existed until 1870, Jews were required to live only in specified neighborhoods called ghettos. In the 19th and (before the end of World War II) 20th centuries, the Roman Catholic Church adhered to a distinction between "good antisemitism" and "bad antisemitism". The "bad" kind promoted hatred of Jews because of their descent. This was considered un-Christian because the Christian message was intended for all of humanity regardless of ethnicity; anyone could become a Christian. The "good" kind criticized alleged Jewish conspiracies to control newspapers, banks, and other institutions, to care only about accumulation of wealth, etc.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: gooqa-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.4078

Training Details

Training Dataset

Unnamed Dataset

• Size: 44,286 training samples
• Columns: question, context, and negative
• Approximate statistics based on the first 1000 samples:

  	question	context	negative
  type	string	string	string
  details	min: 6 tokens mean: 14.5 tokens max: 43 tokens	min: 31 tokens mean: 148.52 tokens max: 256 tokens	min: 28 tokens mean: 149.53 tokens max: 256 tokens

• Samples:

  question	context	negative
  Aside from lodging, what amenities are often offered at inns?	Inns are buildings where travellers can seek lodging and, usually, food and drink. They are typically located in the country or along a highway. In Europe, they possibly first sprang up when the Romans built a system of roads two millennia ago.[citation needed] Some inns in Europe are several centuries old. In addition to providing for the needs of travellers, inns traditionally acted as community gathering places.	The Inns of Court and Inns of Chancery in London started as ordinary inns where barristers met to do business, but became institutions of the legal profession in England and Wales.
  The results of which battle allowed the British to blockade French coasts?	After returning from Egypt, Napoleon engineered a coup in November 1799 and became First Consul of the Republic. Another victory over the Austrians at the Battle of Marengo in 1800 secured his political power. With the Concordat of 1801, Napoleon restored the religious privileges of the Catholic Church while keeping the lands seized by the Revolution. The state continued to nominate the bishops and to control church finances. He extended his political control over France until the Senate declared him Emperor of the French in 1804, launching the French Empire. Intractable differences with the British meant that the French were facing a Third Coalition by 1805. Napoleon shattered this coalition with decisive victories in the Ulm Campaign and a historic triumph at the Battle of Austerlitz, which led to the elimination of the Holy Roman Empire. In October 1805, however, a Franco-Spanish fleet was destroyed at the Battle of Trafalgar, allowing Britain to impose a naval blockade of the Frenc...	By 17 May, the main Austrian army under Charles had arrived on the Marchfeld. Charles kept the bulk of his troops several miles away from the river bank in hopes of concentrating them at the point where Napoleon decided to cross. On 21 May, the French made their first major effort to cross the Danube, precipitating the Battle of Aspern-Essling. The Austrians enjoyed a comfortable numerical superiority over the French throughout the battle; on the first day, Charles disposed of 110,000 soldiers against only 31,000 commanded by Napoleon. By the second day, reinforcements had boosted French numbers up to 70,000. The battle was characterized by a vicious back-and-forth struggle for the two villages of Aspern and Essling, the focal points of the French bridgehead. By the end of the fighting, the French had lost Aspern but still controlled Essling. A sustained Austrian artillery bombardment eventually convinced Napoleon to withdraw his forces back onto Lobau Island. Both sides inflicted abou...
  What is an example of bad treatment causing resistance?	Inappropriate antibiotic treatment and overuse of antibiotics have contributed to the emergence of antibiotic-resistant bacteria. Self prescription of antibiotics is an example of misuse. Many antibiotics are frequently prescribed to treat symptoms or diseases that do not respond to antibiotics or that are likely to resolve without treatment. Also, incorrect or suboptimal antibiotics are prescribed for certain bacterial infections. The overuse of antibiotics, like penicillin and erythromycin, has been associated with emerging antibiotic resistance since the 1950s. Widespread usage of antibiotics in hospitals has also been associated with increases in bacterial strains and species that no longer respond to treatment with the most common antibiotics.	Possible improvements include clarification of clinical trial regulations by FDA. Furthermore, appropriate economic incentives could persuade pharmaceutical companies to invest in this endeavor. Antibiotic Development to Advance Patient Treatment (ADAPT) Act aims to fast track the drug development to combat the growing threat of 'superbugs'. Under this Act, FDA can approve antibiotics and antifungals treating life-threatening infections based on smaller clinical trials. The CDC will monitor the use of antibiotics and the emerging resistance, and publish the data. The FDA antibiotics labeling process, 'Susceptibility Test Interpretive Criteria for Microbial Organisms' or 'breakpoints', will provide accurate data to healthcare professionals. According to Allan Coukell, senior director for health programs at The Pew Charitable Trusts, "By allowing drug developers to rely on smaller datasets, and clarifying FDA's authority to tolerate a higher level of uncertainty for these drugs when maki...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 5,000 evaluation samples
• Columns: question, context, and negative_1
• Approximate statistics based on the first 1000 samples:

  	question	context	negative_1
  type	string	string	string
  details	min: 7 tokens mean: 14.48 tokens max: 52 tokens	min: 28 tokens mean: 150.06 tokens max: 256 tokens	min: 28 tokens mean: 147.16 tokens max: 256 tokens

• Samples:

  question	context	negative_1
  What is the name for the collection of formal rules in Catholicism?	In the Catholic Church, canon law is the system of laws and legal principles made and enforced by the Church's hierarchical authorities to regulate its external organization and government and to order and direct the activities of Catholics toward the mission of the Church.	In the Catholic Church, canon law is the system of laws and legal principles made and enforced by the Church's hierarchical authorities to regulate its external organization and government and to order and direct the activities of Catholics toward the mission of the Church.
  Where is Mount Tangjia?	The most precarious of these quake-lakes was the one located in the extremely difficult terrain at Mount Tangjia in Beichuan County, Sichuan, accessible only by foot or air; an Mi-26T heavy lift helicopter belonging to the China Flying Dragon Special Aviation Company was used to bring heavy earthmoving tractors to the affected location. This operation was coupled with the work done by PLAAF Mi-17 helicopters bringing in PLA engineering corps, explosive specialists and other personnel to join 1,200 soldiers who arrived on site by foot. Five tons of fuel to operate the machinery was airlifted to the site, where a sluice was constructed to allow the safe discharge of the bottlenecked water. Downstream, more than 200,000 people were evacuated from Mianyang by June 1 in anticipation of the dam bursting.	Tibet was once a strong power contemporaneous with Tang China (618–907). Until the Tibetan Empire's collapse in the 9th century, it was the Tang's major rival in dominating Inner Asia. The Yarlung rulers of Tibet also signed various peace treaties with the Tang, culminating in a treaty in 821 that fixed the borders between Tibet and China.
  What century did Aretas III Philhellene reign?	The Nabatean Kingdom was an Arab state located between the Sinai Peninsula and the Arabian Peninsula. Its capital was the city of Petra, an important trading city on the incense route. The Nabateans resisted the attacks of Antigonous and were allies of the Hasmoneans in their struggle against the Seleucids, but later fought against Herod the great. The hellenization of the Nabateans accured relatively late in comparison to the surrounding regions. Nabatean material culture does not show any Greek influence until the reign of Aretas III Philhellene in the 1st century BCE. Aretas captured Damascus and built the Petra pool complex and gardens in the Hellenistic style. Though the Nabateans originally worshipped their traditional gods in symbolic form such as stone blocks or pillars, during the Hellenistic period they began to identify their gods with Greek gods and depict them in figurative forms influenced by Greek sculpture. Nabatean art shows Greek influences and paintings have been fou...	The Nabatean Kingdom was an Arab state located between the Sinai Peninsula and the Arabian Peninsula. Its capital was the city of Petra, an important trading city on the incense route. The Nabateans resisted the attacks of Antigonous and were allies of the Hasmoneans in their struggle against the Seleucids, but later fought against Herod the great. The hellenization of the Nabateans accured relatively late in comparison to the surrounding regions. Nabatean material culture does not show any Greek influence until the reign of Aretas III Philhellene in the 1st century BCE. Aretas captured Damascus and built the Petra pool complex and gardens in the Hellenistic style. Though the Nabateans originally worshipped their traditional gods in symbolic form such as stone blocks or pillars, during the Hellenistic period they began to identify their gods with Greek gods and depict them in figurative forms influenced by Greek sculpture. Nabatean art shows Greek influences and paintings have been fou...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• num_train_epochs: 4
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooqa-dev_cosine_accuracy
-1	-1	-	-	0.3294
0.5780	100	0.5447	0.8324	0.3926
1.1561	200	0.4731	0.8196	0.3952
1.7341	300	0.3978	0.8026	0.4026
2.3121	400	0.3496	0.7996	0.4046
2.8902	500	0.3252	0.7948	0.4040
3.4682	600	0.2958	0.7955	0.4070
-1	-1	-	-	0.4078

Framework Versions

• Python: 3.11.0
• Sentence Transformers: 4.0.1
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}