ayushexel/embed-all-MiniLM-L6-v2-squad-3-epochs

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ayushexel/embed-all-MiniLM-L6-v2-squad-3-epochs")
# Run inference
sentences = [
    'What did Tom Parsons consider as the risk factor for strong future quakes?',
    'In a United States Geological Survey (USGS) study, preliminary rupture models of the earthquake indicated displacement of up to 9 meters along a fault approximately 240 km long by 20 km deep. The earthquake generated deformations of the surface greater than 3 meters and increased the stress (and probability of occurrence of future events) at the northeastern and southwestern ends of the fault. On May 20, USGS seismologist Tom Parsons warned that there is "high risk" of a major M>7 aftershock over the next weeks or months.',
    'The earthquake was the worst to strike the Sichuan area in over 30 years. Following the quake, experts and the general public sought information on whether or not the earthquake could have been predicted in advance, and whether or not studying statistics related to the quake could result in better prediction of earthquakes in the future. Earthquake prediction is not yet established science; there was no consensus within the scientific community that earthquake "prediction" is possible.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: gooqa-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.4034

Training Details

Training Dataset

Unnamed Dataset

• Size: 44,284 training samples
• Columns: question, context, and negative
• Approximate statistics based on the first 1000 samples:

  	question	context	negative
  type	string	string	string
  details	min: 6 tokens mean: 14.48 tokens max: 34 tokens	min: 28 tokens mean: 145.96 tokens max: 256 tokens	min: 28 tokens mean: 151.27 tokens max: 256 tokens

• Samples:

  question	context	negative
  Some foods that are okay for people to eat are what to dogs?	A number of common human foods and household ingestibles are toxic to dogs, including chocolate solids (theobromine poisoning), onion and garlic (thiosulphate, sulfoxide or disulfide poisoning), grapes and raisins, macadamia nuts, xylitol, as well as various plants and other potentially ingested materials. The nicotine in tobacco can also be dangerous. Dogs can get it by scavenging in garbage or ashtrays; eating cigars and cigarettes. Signs can be vomiting of large amounts (e.g., from eating cigar butts) or diarrhea. Some other signs are abdominal pain, loss of coordination, collapse, or death. Dogs are highly susceptible to theobromine poisoning, typically from ingestion of chocolate. Theobromine is toxic to dogs because, although the dog's metabolism is capable of breaking down the chemical, the process is so slow that even small amounts of chocolate can be fatal, especially dark chocolate.	Greek cuisine is characteristic of the healthy Mediterranean diet, which is epitomized by dishes of Crete. Greek cuisine incorporates fresh ingredients into a variety of local dishes such as moussaka, stifado, Greek salad, fasolada, spanakopita and souvlaki. Some dishes can be traced back to ancient Greece like skordalia (a thick purée of walnuts, almonds, crushed garlic and olive oil), lentil soup, retsina (white or rosé wine sealed with pine resin) and pasteli (candy bar with sesame seeds baked with honey). Throughout Greece people often enjoy eating from small dishes such as meze with various dips such as tzatziki, grilled octopus and small fish, feta cheese, dolmades (rice, currants and pine kernels wrapped in vine leaves), various pulses, olives and cheese. Olive oil is added to almost every dish.
  Which city is often referred to as Australia's garden city?	Melbourne is often referred to as Australia's garden city, and the state of Victoria was once known as the garden state. There is an abundance of parks and gardens in Melbourne, many close to the CBD with a variety of common and rare plant species amid landscaped vistas, pedestrian pathways and tree-lined avenues. Melbourne's parks are often considered the best public parks in all of Australia's major cities. There are also many parks in the surrounding suburbs of Melbourne, such as in the municipalities of Stonnington, Boroondara and Port Phillip, south east of the central business district. The extensive area covered by urban Melbourne is formally divided into hundreds of suburbs (for addressing and postal purposes), and administered as local government areas 31 of which are located within the metropolitan area.	Melbourne is typical of Australian capital cities in that after the turn of the 20th century, it expanded with the underlying notion of a 'quarter acre home and garden' for every family, often referred to locally as the Australian Dream. This, coupled with the popularity of the private automobile after 1945, led to the auto-centric urban structure now present today in the middle and outer suburbs. Much of metropolitan Melbourne is accordingly characterised by low density sprawl, whilst its inner city areas feature predominantly medium-density, transit-oriented urban forms. The city centre, Docklands, St. Kilda Road and Southbank areas feature high-density forms.
  How much is Israel's water technology industry worth?	Israel is one of the world's technological leaders in water technology. In 2011, its water technology industry was worth around $2 billion a year with annual exports of products and services in the tens of millions of dollars. The ongoing shortage of water in the country has spurred innovation in water conservation techniques, and a substantial agricultural modernization, drip irrigation, was invented in Israel. Israel is also at the technological forefront of desalination and water recycling. The Ashkelon seawater reverse osmosis (SWRO) plant, the largest in the world, was voted 'Desalination Plant of the Year' in the Global Water Awards in 2006. Israel hosts an annual Water Technology Exhibition and Conference (WaTec) that attracts thousands of people from across the world. By 2014, Israel's desalination programs provided roughly 35% of Israel's drinking water and it is expected to supply 40% by 2015 and 70% by 2050. As of May 29, 2015 more than 50 percent of the water for Israeli ho...	Israel is a leading country in the development of solar energy. Israel is a global leader in water conservation and geothermal energy, and its development of cutting-edge technologies in software, communications and the life sciences have evoked comparisons with Silicon Valley. According to the OECD, Israel is also ranked 1st in the world in expenditure on Research and Development (R&D) as a percentage of GDP. Intel and Microsoft built their first overseas research and development centers in Israel, and other high-tech multi-national corporations, such as IBM, Google, Apple, HP, Cisco Systems, and Motorola, have opened R&D facilities in the country.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 5,000 evaluation samples
• Columns: question, context, and negative_1
• Approximate statistics based on the first 1000 samples:

  	question	context	negative_1
  type	string	string	string
  details	min: 3 tokens mean: 14.64 tokens max: 52 tokens	min: 32 tokens mean: 146.0 tokens max: 256 tokens	min: 29 tokens mean: 144.8 tokens max: 256 tokens

• Samples:

  question	context	negative_1
  What language form differs in the amount of vowel reduction?	Catalan has inherited the typical vowel system of Vulgar Latin, with seven stressed phonemes: /a ɛ e i ɔ o u/, a common feature in Western Romance, except Spanish. Balearic has also instances of stressed /ə/. Dialects differ in the different degrees of vowel reduction, and the incidence of the pair /ɛ e/.	In Central Catalan, unstressed vowels reduce to three: /a e ɛ/ > [ə]; /o ɔ u/ > [u]; /i/ remains distinct. The other dialects have different vowel reduction processes (see the section pronunciation of dialects in this article).
  Who released the first collection of Chopin's works?	Chopin's original publishers included Maurice Schlesinger and Camille Pleyel. His works soon began to appear in popular 19th-century piano anthologies. The first collected edition was by Breitkopf & Härtel (1878–1902). Among modern scholarly editions of Chopin's works are the version under the name of Paderewski published between 1937 and 1966 and the more recent Polish "National Edition", edited by Jan Ekier, both of which contain detailed explanations and discussions regarding choices and sources.	Possibly the first venture into fictional treatments of Chopin's life was a fanciful operatic version of some of its events. Chopin was written by Giacomo Orefice and produced in Milan in 1901. All the music is derived from that of Chopin.
  How many students in New York partcipate in higher education?	Over 600,000 students are enrolled in New York City's over 120 higher education institutions, the highest number of any city in the United States, including over half million in the City University of New York (CUNY) system alone in 2014. In 2005, three out of five Manhattan residents were college graduates, and one out of four had a postgraduate degree, forming one of the highest concentrations of highly educated people in any American city. New York City is home to such notable private universities as Barnard College, Columbia University, Cooper Union, Fordham University, New York University, New York Institute of Technology, Pace University, and Yeshiva University. The public CUNY system is one of the largest universities in the nation, comprising 24 institutions across all five boroughs: senior colleges, community colleges, and other graduate/professional schools. The public State University of New York (SUNY) system also serves New York City, as well as the rest of the state. The ...	Over 600,000 students are enrolled in New York City's over 120 higher education institutions, the highest number of any city in the United States, including over half million in the City University of New York (CUNY) system alone in 2014. In 2005, three out of five Manhattan residents were college graduates, and one out of four had a postgraduate degree, forming one of the highest concentrations of highly educated people in any American city. New York City is home to such notable private universities as Barnard College, Columbia University, Cooper Union, Fordham University, New York University, New York Institute of Technology, Pace University, and Yeshiva University. The public CUNY system is one of the largest universities in the nation, comprising 24 institutions across all five boroughs: senior colleges, community colleges, and other graduate/professional schools. The public State University of New York (SUNY) system also serves New York City, as well as the rest of the state. The ...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooqa-dev_cosine_accuracy
-1	-1	-	-	0.3264
0.2890	100	0.4336	0.7852	0.3846
0.5780	200	0.3867	0.7628	0.3992
0.8671	300	0.3847	0.7571	0.3930
1.1561	400	0.3119	0.7450	0.4062
1.4451	500	0.2727	0.7444	0.4076
1.7341	600	0.2769	0.7392	0.4052
2.0231	700	0.269	0.7371	0.4032
2.3121	800	0.2084	0.7373	0.4010
2.6012	900	0.2095	0.7354	0.4032
2.8902	1000	0.213	0.7354	0.4096
-1	-1	-	-	0.4034

Framework Versions

• Python: 3.11.0
• Sentence Transformers: 4.0.1
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}