VinitT/Embeddings-Trivia

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2 on the trivia-qa-triplet dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • trivia-qa-triplet
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'In the RAF, what is the rank immediately above Squadron Leader?',
    'Squadron leader Squadron leader Squadron leader (Sqn Ldr in the RAF ; SQNLDR in the RAAF and RNZAF; formerly sometimes S/L in all services) is a commissioned rank in the Royal Air Force and the air forces of many countries which have historical British influence. It is also sometimes used as the English translation of an equivalent rank in countries which have a non-English air force-specific rank structure. An air force squadron leader ranks above flight lieutenant and immediately below wing commander and it is the most junior of the senior officer ranks. The air force rank of squadron leader has a',
    'Squadron leader RAF used major as the equivalent rank to squadron leader. Royal Naval Air Service lieutenant-commanders and Royal Flying Corps majors on 31 March 1918 became RAF majors on 1 April 1918. On 31 August 1919, the RAF rank of major was superseded by squadron leader which has remained in continuous usage ever since. Promotion to squadron leader is strictly on merit, and requires the individual to be appointed to a Career Commission, which will see them remain in the RAF until retirement or voluntary resignation. Before the Second World War, a squadron leader commanded a squadron of aircraft. Today, however,',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5605, 0.4394],
#         [0.5605, 1.0000, 0.5768],
#         [0.4394, 0.5768, 1.0000]])

Evaluation

Metrics

Triplet

• Dataset: trivia_qa_eval
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.834

Training Details

Training Dataset

trivia-qa-triplet

• Dataset: trivia-qa-triplet at bfe9460
• Size: 59,315 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 8 tokens mean: 20.08 tokens max: 71 tokens	min: 114 tokens mean: 139.64 tokens max: 226 tokens	min: 113 tokens mean: 138.81 tokens max: 256 tokens

• Samples:

  anchor	positive	negative
  "Sirius otherwise known as the ""Dog Star"" can be found in which constellation?"	Sirius in fiction below)—and even if such were eventually discovered, with an estimated age of 230 million years the system is too young to have fostered the development of life or a complex biosphere. Sirius AB is the alpha star of the constellation Canis Major (the great dog, sometimes styled as Orion's hunting dog), whence its cognomen "the dog star". The most commonly used proper name of this star comes through the Latin "Sirius", from the Greek Σείριος ("Seirios", "glowing" or "scorcher"). The ancient Greeks observed that the appearance of Sirius heralded the hot and dry "dog days" of summer, and feared that	Dog days "Seírios", "Scorcher"), Sothis (, "Sôthis", a transcription of Egyptian "Spdt"), and the Dog Star (, "Kúōn"). The last name reflects the way Sirius follows the constellation Orion into the night sky. Sirius is by far the brightest proper star in the night sky, which caused ancient astronomers to take note of it around the world. In Egypt, its return to the night sky became known as a precursor to the annual flooding of the Nile and was worshipped as the goddess Sopdet. In Greece, it became known as the precursor of the unpleasantly hot phase of the summer. Greek poets
  Which Scottish town is the administrative centre of the Highland Region?	Highland Scottish All operations are now part of the Stagecoach Group. Highland Scottish Highland Scottish Omnibuses Ltd was formed as a bus operating subsidiary of the Scottish Transport Group in June 1985 from Highland Omnibuses Ltd, and operated until October 1995 when the company was split into two - Highland Bus & Coach and Highland Country Buses. The companies have since remerged and operate today as Highland Country Buses. From its head office in Seafield Road, Inverness, Highland Scottish operated over the massive geographical, but sparsely populated, area of the Highland region of north west Scotland. Highland Scottish was the largest operator	Highland Scottish in the centre of its 'Highland Country' logo. In January 1996 Highland Country Buses was bought by National Express for £1.8m. Highland Bus & Coach, being the smaller of the two operators, continued to operate with the image its predecessor adopted. The two companies continued to exist under separate ownership until August 1998 when Rapson's bought Highland Country Buses back from National Express for £4m - £2.2m more than Rapson's originally sold the company for. Highland Country Buses is now a wholly owned subsidiary of Rapson's Coaches, and covers the operating area that Highland Scottish had on privatisation. The company
  Which brand of coffee is named after a hotel in Nashville, Tennessee?	History of Nashville, Tennessee proprietor thereof, had served a special blend of coffee at the hotel's restaurant, and after drinking a cup of this coffee, Roosevelt proclaimed it "good to the last drop!" Cheek subsequently sold the blend to General Foods and to this day, Maxwell House coffee is enjoyed by millions. In 1913 Nashville was the last of several major cities in Tennessee to adopt a commission form of government, with all members of a small commission elected at-large. Compared to single-member districts, this change resulted in further limiting the political power of any African Americans who were able to vote, as their	Renaissance Nashville Hotel Renaissance Nashville Hotel The Renaissance Nashville Hotel is a hotel in Nashville, Tennessee. The building is 385 feet high with 31 floors. The hotel is physically connected to the Nashville Convention Center and is its anchor hotel. The hotel contains 649 rooms, 24 suites, 25 meeting rooms with 31,000 sq ft of meeting space, and 2 concierge levels including a Starbucks coffee shop, 2 lounges, and a full service restaurant. One of the lounges is located in an enclosed bridge walkway, spanning above Commerce Street, which connects the hotel to a parking garage across the street. This walkway was severely

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

trivia-qa-triplet

• Dataset: trivia-qa-triplet at bfe9460
• Size: 1,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 8 tokens mean: 20.18 tokens max: 103 tokens	min: 111 tokens mean: 139.08 tokens max: 225 tokens	min: 113 tokens mean: 138.94 tokens max: 256 tokens

• Samples:

  anchor	positive	negative
  What African country is projected to pass the United States in population by the year 2055?	African immigration to the United States entering the United States. It is estimated that the current population of African immigrants to the United States is about 2.1 million. According to the Migration Policy Institute, as of 2009 two-thirds of the African immigrants were from either East or West Africa. Countries with the most immigrants to the U.S. are Nigeria, Egypt, Ethiopia, Ghana, South Africa, Somalia, Eritrea, and Kenya. Seventy five percent (75%) of the African immigrants to the U.S. come from 12 of the 55 countries, namely Nigeria, Egypt, Ghana, Ethiopia, South Africa, Kenya, Liberia, Somalia, Morocco, Cape Verde, Sierra Leone and Sudan (including what is	African immigration to the United States African countries due to many skilled hard-working Africans leaving Africa to seek their economic fortunes in the U.S. mainly and elsewhere. One major factor that contributes to migration from Africa to the United States is labor opportunities. It has been relatively easier for African immigrants for Africans with advanced education to leave and enter international labor markets. In addition, many Africans move to the United States for advanced training. For example, doctors from different African nations would move to the U.S. in order to gain more economic opportunities compared to their home country. However, as more Africans emigrate to the
  Which is the largest species of the turtle family?	Leatherback sea turtle Leatherback sea turtle The leatherback sea turtle ("Dermochelys coriacea"), sometimes called the lute turtle or leathery turtle or simply the luth, is the largest of all living turtles and is the fourth-heaviest modern reptile behind three crocodilians. It is the only living species in the genus Dermochelys and family Dermochelyidae. It can easily be differentiated from other modern sea turtles by its lack of a bony shell, hence the name. Instead, its carapace is covered by skin and oily flesh. "Dermochelys" is the only extant genus of the family Dermochelyidae. "Dermochelys coriacea" is the only species in genus "Dermochelys". The	Okavango mud turtle Okavango mud turtle The Okavango mud turtle (Okavango terrapin) ("Pelusios bechuanicus") is a species of turtle in the family Pelomedusidae endemic to Africa. It is found in Angola, Botswana, the Democratic Republic of the Congo, Namibia (Caprivi), Zambia, and Zimbabwe. Found in central Africa, central Angola, northeastern Namibia, northern Botswana, Zimbabwe, and Zambia The Okavango mud turtle is largest species of the genus "Pelusios". The carapace is oval and elongated, with a pronounced dome, and is evenly rounded at the edges which allows the turtle to appear as a smooth rock. The carapace is very dark, often almost black, and
  How many gallons of beer are in an English barrel?	Barrel gallon for liquids (the corn gallon of 268.8 cubic inches for solids). In Britain, the wine gallon was replaced by the imperial gallon. The tierce later became the petrol barrel. The tun was originally 256 gallons, which explains from where the quarter, 8 bushels or 64 (wine) gallons, comes. Although it is common to refer to draught beer containers of any size as barrels, in the UK this is strictly correct only if the container holds 36 imperial gallons. The terms "keg" and "cask" refer to containers of any size, the distinction being that kegs are used for beers intended	Beer Barrel Polecats Percy Pomeroy (Eddie Laughton), and work together to flee the prison. They are ultimately captured, and sent to solitary confinement. After nearly half a century later, the graying trio are finally released as senior citizens, in which Curly quips upon leaving "You know what I'm-a gonna do? I'm gonna get myself a tall, big, beautiful bottle of beer!" Moe and Larry become irate and throw Curly back into the jail, leaving him there. The title "Beer Barrel Polecats" is a pun of the song "Beer Barrel Polka". The idea of producing and selling their own beer during Prohibition was borrowed

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• learning_rate: 2e-05
• weight_decay: 0.01
• num_train_epochs: 4
• warmup_ratio: 0.1
• warmup_steps: 0.1
• fp16: True
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: 0.1
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	trivia_qa_eval_cosine_accuracy
0.0011	1	0.7126	-	-
0.0539	50	0.7164	-	-
0.1079	100	0.7126	-	-
0.1618	150	0.6888	-	-
0.2157	200	0.6802	-	-
0.2697	250	0.6422	-	-
0.3236	300	0.6562	-	-
0.3776	350	0.6356	-	-
0.4315	400	0.6532	-	-
0.4854	450	0.6106	-	-
0.5394	500	0.6104	0.5472	0.7970
0.5933	550	0.6301	-	-
0.6472	600	0.6259	-	-
0.7012	650	0.5759	-	-
0.7551	700	0.6089	-	-
0.8091	750	0.5835	-	-
0.8630	800	0.5890	-	-
0.9169	850	0.5577	-	-
0.9709	900	0.5569	-	-
1.0248	950	0.5427	-	-
1.0787	1000	0.4698	0.5046	0.8190
1.1327	1050	0.4662	-	-
1.1866	1100	0.4634	-	-
1.2406	1150	0.4597	-	-
1.2945	1200	0.4585	-	-
1.3484	1250	0.5140	-	-
1.4024	1300	0.4542	-	-
1.4563	1350	0.4579	-	-
1.5102	1400	0.4910	-	-
1.5642	1450	0.5067	-	-
1.6181	1500	0.4800	0.4875	0.8300
1.6721	1550	0.4638	-	-
1.7260	1600	0.4760	-	-
1.7799	1650	0.4699	-	-
1.8339	1700	0.4912	-	-
1.8878	1750	0.4726	-	-
1.9417	1800	0.4764	-	-
1.9957	1850	0.4802	-	-
2.0496	1900	0.3941	-	-
2.1036	1950	0.3991	-	-
2.1575	2000	0.4114	0.4734	0.838
2.2114	2050	0.3981	-	-
2.2654	2100	0.4023	-	-
2.3193	2150	0.3932	-	-
2.3732	2200	0.3887	-	-
2.4272	2250	0.3894	-	-
2.4811	2300	0.3858	-	-
2.5351	2350	0.3907	-	-
2.5890	2400	0.3934	-	-
2.6429	2450	0.3871	-	-
2.6969	2500	0.3763	0.4681	0.8310
2.7508	2550	0.3997	-	-
2.8047	2600	0.3941	-	-
2.8587	2650	0.3884	-	-
2.9126	2700	0.3771	-	-
2.9666	2750	0.4168	-	-
3.0205	2800	0.3722	-	-
3.0744	2850	0.3565	-	-
3.1284	2900	0.3499	-	-
3.1823	2950	0.3428	-	-
3.2362	3000	0.3583	0.4669	0.8320
3.2902	3050	0.3444	-	-
3.3441	3100	0.3252	-	-
3.3981	3150	0.3563	-	-
3.4520	3200	0.3465	-	-
3.5059	3250	0.3328	-	-
3.5599	3300	0.3438	-	-
3.6138	3350	0.3330	-	-
3.6677	3400	0.3567	-	-
3.7217	3450	0.3462	-	-
3.7756	3500	0.3435	0.4639	0.8340
3.8296	3550	0.3532	-	-
3.8835	3600	0.3480	-	-
3.9374	3650	0.3361	-	-
3.9914	3700	0.3628	-	-

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.2
• Transformers: 5.0.0
• PyTorch: 2.9.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}