ayushexel/emb-all-MiniLM-L6-v2-squad-2-epochs

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ayushexel/emb-all-MiniLM-L6-v2-squad-2-epochs")
# Run inference
sentences = [
    'In what year did CBS sell CBS Songs?',
    'In 1986, CBS sold its music publishing arm, CBS Songs, to Stephen Swid, Martin Bandier, and Charles Koppelman for $125 million making it the foundation of their SBK Entertainment.',
    'On November 17, 1987, SCA acquired CBS Records, which hosted such acts as Michael Jackson, for US$2 billion. CBS Inc., now CBS Corporation, retained the rights to the CBS name for music recordings but granted Sony a temporary license to use the CBS name. CBS Corporation founded a new CBS Records in 2006, which is distributed by Sony through its RED subsidiary.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Dataset: gooqa-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.4062

Training Details

Training Dataset

Unnamed Dataset

• Size: 44,286 training samples
• Columns: question, context, and negative
• Approximate statistics based on the first 1000 samples:

  	question	context	negative
  type	string	string	string
  details	min: 7 tokens mean: 14.66 tokens max: 43 tokens	min: 28 tokens mean: 145.37 tokens max: 256 tokens	min: 29 tokens mean: 149.74 tokens max: 256 tokens

• Samples:

  question	context	negative
  Who was the first governor of Saint Helena?	In 1657, Oliver Cromwell granted the English East India Company a charter to govern Saint Helena and the following year the company decided to fortify the island and colonise it with planters. The first governor, Captain John Dutton, arrived in 1659, making Saint Helena one of Britain's oldest colonies outside North America and the Caribbean. A fort and houses were built. After the Restoration of the English monarchy in 1660, the East India Company received a royal charter giving it the sole right to fortify and colonise the island. The fort was renamed James Fort and the town Jamestown, in honour of the Duke of York, later James II of England.	In 2012, the government of St. Helena funded the creation of the St. Helena Human Rights Action Plan 2012-2015. Work is being done under this action plan, including publishing awareness-raising articles in local newspapers, providing support for members of the public with human rights queries, and extending several UN Conventions on human rights to St. Helena.
  When was the Chapultepec Zoo opened?	Mexico City has three zoos. Chapultepec Zoo, the San Juan de Aragon Zoo and Los Coyotes Zoo. Chapultepec Zoo is located in the first section of Chapultepec Park in the Miguel Hidalgo. It was opened in 1924. Visitors can see about 243 specimens of different species including kangaroos, giant panda, gorillas, caracal, hyena, hippos, jaguar, giraffe, lemur, lion, among others. Zoo San Juan de Aragon is near the San Juan de Aragon Park in the Gustavo A. Madero. In this zoo, opened in 1964, there are species that are in danger of extinction such as the jaguar and the Mexican wolf. Other guests are the golden eagle, pronghorn, bighorn sheep, caracara, zebras, African elephant, macaw, hippo, among others. Zoo Los Coyotes is a 27.68-acre (11.2 ha) zoo located south of Mexico City in the Coyoacan. It was inaugurated on February 2, 1999. It has more than 301 specimens of 51 species of wild native or endemic fauna from the Mexico City. You can admire eagles, ajolotes, coyotes, macaws, bobcats, Me...	Chapultepec Park houses the Chapultepec Castle, now a museum on a hill that overlooks the park and its numerous museums, monuments and the national zoo and the National Museum of Anthropology (which houses the Aztec Calendar Stone). Another piece of architecture is the Fine Arts Palace, a white marble theatre/museum whose weight is such that it has gradually been sinking into the soft ground below. Its construction began during the presidency of Porfirio Díaz and ended in 1934, after being interrupted by the Mexican Revolution in the 1920s. The Plaza of the Three Cultures in the Tlatelolco neighbourhood, and the shrine and Basilicas of Our Lady of Guadalupe are also important sites. There is a double-decker bus, known as the "Turibus", that circles most of these sites, and has timed audio describing the sites in multiple languages as they are passed.
  Which season three contestant on American Idol was the recipient of reported death threats?	Much media attention on the season had been focused on the three black singers, Fantasia Barrino, LaToya London, and Jennifer Hudson, dubbed the Three Divas. All three unexpectedly landed on the bottom three on the top seven result show, with Hudson controversially eliminated. Elton John, who was one of the mentors that season, called the results of the votes "incredibly racist". The prolonged stays of John Stevens and Jasmine Trias in the finals, despite negative comments from the judges, had aroused resentment, so much so that John Stevens reportedly received a death threat, which he dismissed as a joke 'blown out of proportion'.	The winner receives a record deal with a major label, which may be for up to six albums, and secures a management contract with American Idol-affiliated 19 Management (which has the right of first refusal to sign all contestants), as well as various lucrative contracts. All winners prior to season nine reportedly earned at least $1 million in their first year as winner. All the runners-up of the first ten seasons, as well as some of other finalists, have also received record deals with major labels. However, starting in season 11, the runner-up may only be guaranteed a single-only deal. BMG/Sony (seasons 1–9) and UMG (season 10–) had the right of first refusal to sign contestants for three months after the season's finale. Starting in the fourteenth season, the winner was signed with Big Machine Records. Prominent music mogul Clive Davis also produced some of the selected contestants' albums, such as Kelly Clarkson, Clay Aiken, Fantasia Barrino and Diana DeGarmo. All top 10 (11 in seas...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 5,000 evaluation samples
• Columns: question, context, and negative_1
• Approximate statistics based on the first 1000 samples:

  	question	context	negative_1
  type	string	string	string
  details	min: 7 tokens mean: 14.59 tokens max: 52 tokens	min: 28 tokens mean: 146.96 tokens max: 256 tokens	min: 28 tokens mean: 143.9 tokens max: 256 tokens

• Samples:

  question	context	negative_1
  Stain-resistant fabrics are coated with chemicals to reduce what?	The world of clothing is always changing, as new cultural influences meet technological innovations. Researchers in scientific labs have been developing prototypes for fabrics that can serve functional purposes well beyond their traditional roles, for example, clothes that can automatically adjust their temperature, repel bullets, project images, and generate electricity. Some practical advances already available to consumers are bullet-resistant garments made with kevlar and stain-resistant fabrics that are coated with chemical mixtures that reduce the absorption of liquids.	The world of clothing is always changing, as new cultural influences meet technological innovations. Researchers in scientific labs have been developing prototypes for fabrics that can serve functional purposes well beyond their traditional roles, for example, clothes that can automatically adjust their temperature, repel bullets, project images, and generate electricity. Some practical advances already available to consumers are bullet-resistant garments made with kevlar and stain-resistant fabrics that are coated with chemical mixtures that reduce the absorption of liquids.
  What explorer led the Nimrod Expedition into Antarctica?	During the Nimrod Expedition led by Ernest Shackleton in 1907, parties led by Edgeworth David became the first to climb Mount Erebus and to reach the South Magnetic Pole. Douglas Mawson, who assumed the leadership of the Magnetic Pole party on their perilous return, went on to lead several expeditions until retiring in 1931. In addition, Shackleton himself and three other members of his expedition made several firsts in December 1908 – February 1909: they were the first humans to traverse the Ross Ice Shelf, the first to traverse the Transantarctic Mountains (via the Beardmore Glacier), and the first to set foot on the South Polar Plateau. An expedition led by Norwegian polar explorer Roald Amundsen from the ship Fram became the first to reach the geographic South Pole on 14 December 1911, using a route from the Bay of Whales and up the Axel Heiberg Glacier. One month later, the doomed Scott Expedition reached the pole.	During the Nimrod Expedition led by Ernest Shackleton in 1907, parties led by Edgeworth David became the first to climb Mount Erebus and to reach the South Magnetic Pole. Douglas Mawson, who assumed the leadership of the Magnetic Pole party on their perilous return, went on to lead several expeditions until retiring in 1931. In addition, Shackleton himself and three other members of his expedition made several firsts in December 1908 – February 1909: they were the first humans to traverse the Ross Ice Shelf, the first to traverse the Transantarctic Mountains (via the Beardmore Glacier), and the first to set foot on the South Polar Plateau. An expedition led by Norwegian polar explorer Roald Amundsen from the ship Fram became the first to reach the geographic South Pole on 14 December 1911, using a route from the Bay of Whales and up the Axel Heiberg Glacier. One month later, the doomed Scott Expedition reached the pole.
  When was the first female ascent on Mont Blanc?	The first British Mont Blanc ascent was in 1788; the first female ascent in 1819. By the mid-1850s Swiss mountaineers had ascended most of the peaks and were eagerly sought as mountain guides. Edward Whymper reached the top of the Matterhorn in 1865 (after seven attempts), and in 1938 the last of the six great north faces of the Alps was climbed with the first ascent of the Eiger Nordwand (north face of the Eiger).	The first British Mont Blanc ascent was in 1788; the first female ascent in 1819. By the mid-1850s Swiss mountaineers had ascended most of the peaks and were eagerly sought as mountain guides. Edward Whymper reached the top of the Matterhorn in 1865 (after seven attempts), and in 1938 the last of the six great north faces of the Alps was climbed with the first ascent of the Eiger Nordwand (north face of the Eiger).

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• num_train_epochs: 2
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	gooqa-dev_cosine_accuracy
-1	-1	-	-	0.3286
0.5780	100	0.5372	0.8373	0.4002
1.1561	200	0.4631	0.8101	0.4034
1.7341	300	0.3993	0.8093	0.4078
-1	-1	-	-	0.4062

Framework Versions

• Python: 3.11.0
• Sentence Transformers: 4.0.1
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.5.2
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}