README.md

metadata

base_model: distilbert/distilbert-base-uncased
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:676193
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: which type of tides have the largest range
    sentences:
      - >-
        Your BMI is based on your height and weight. It's one way to see if
        you're at a healthy weight. Underweight: Your BMI is less than 18.5.
        Healthy weight: Your BMI is 18.5 to 24.9. Overweight: Your BMI is 25 to
        29.9. Obese: Your BMI is 30 or higher. The chart below shows examples of
        body mass indexes. The figure at which your height corresponds with your
        weight is your body mass index.
      - >-
        For example, a slight color change in the test pad for protein may
        indicate a small amount of protein present in the urine whereas a deep
        color change may indicate a large amount. The most frequently performed
        chemical tests using reagent test strips are: 1  Specific gravity.
      - >-
        When the moon is full or new, the gravitational pull of the moon and sun
        are combined. At these times, the high tides are very high and the low
        tides are very low. This is known as a spring high tide. Spring tides
        are especially strong tides (they do not have anything to do with the
        season Spring). They occur when the Earth, the Sun, and the Moon are in
        a line. The gravitational forces of the Moon and the Sun both contribute
        to the tides. Spring tides occur during the full moon and the new moon.
  - source_sentence: what is the mexican hat dance what are the moves
    sentences:
      - >-
        You’ve probably heard about the mis-selling of payment protection
        insurance, the “reclaim PPI” adverts, and the refunds people have
        received. Because of the high payouts, a lot of claims management
        companies have sprung up, trying to earn commissions from claiming
        refunds on behalf of their clients. 
      - >-
        These symptoms could be signs of a bacterial infection, such as strep
        throat. Taking antibiotics won’t help at all if your sore throat is
        caused by viruses, but they’re essential for fighting bacterial
        infections like strep. Strep is the most common bacterial throat
        infection. Although it can occur in adults, strep throat is more common
        in children between ages 5 and 15. Riddle says strep can be harder to
        detect in younger children, because it can cause a runny nose and other
        symptoms that make it seem like a cold. Another fairly common throat
        infection is tonsillitis, which occurs when you have sore, swollen
        tonsils. It’s caused by many of the same viruses and bacteria that cause
        sore throats. If you have frequent bouts of tonsillitis or strep throat,
        you may need surgery (called a tonsillectomy) to have your tonsils
        removed.
      - >-
        Jarabe Tapatio (Mexican Hat Dance) -- April 2010. To learn the dance
        often considered the national dance of Mexico. To learn words from the
        Spanish language and facts about the country of Mexico. 
  - source_sentence: where is murchison location
    sentences:
      - >-
        Share. The cerebral cortex is the layer of the brain often referred to
        as gray matter. The cortex (thin layer of tissue) is gray because nerves
        in this area lack the insulation that makes most other parts of the
        brain appear to be white. The cortex covers the outer portion (1.5mm to
        5mm) of the cerebrum and cerebellum. The portion of the cortex that
        covers the cerebrum is called the cerebral cortex. The cerebral cortex
        consists of folded bulges called gyri that create deep furrows or
        fissures called sulci.
      - >-
        Murchison is a small riverside rural village located on the Goulburn
        River in Victoria, Australia. Murchison is located 167 kilometres from
        Melbourne and is just to the west of the Goulburn Valley Highway between
        Shepparton and Nagambie. The surrounding countryside contains orchards,
        vineyards and dairy farms and also HM Prison Dhurringile. At the 2011
        census, Murchison had a population of 1,047
      - >-
        Medicare beneficiary means an individual who is entitled to benefits
        under medicare part A plan and enrolled under medicare part B plan or
        enrolled in both medicare part A and part B plan and who resides in the
        U.S. Medicare beneficiaries pay deductibles and 20 percent coinsurance
        for most services and equipment. Whenever admitted to a hospital for a
        new spell of illness or benefit period, a beneficiary is entitled to
        another 90 days of Part A coverage. In addition, each Medicare
        beneficiary has a lifetime reserve of 60 days that the beneficiary may
        elect to use toward one or more hospital stays. 42 C.F.R. § 409.61 [a]
        [2]. However, if the beneficiary has elected to apply the 60 reserve
        days to a previous hospital stay, the lifetime reserve is exhausted
  - source_sentence: is hpv a std
    sentences:
      - >-
        HPV is the most common sexually transmitted infection (STI). HPV is a
        different virus than HIV and HSV (herpes). HPV is so common that nearly
        all sexually active men and women get it at some point in their lives.
        There are many different types of HPV. 
      - >-
        Hibiscus plants reach a wide variety of heights due to the diversity of
        the species. Grown as annuals, perennials or shrubs, the height range
        includes dwarf varieties as well taller plants that grow up to 15 feet
        tall. Red leaf hibiscus (H. acetosella) is an annual tropical shrub that
        grows to a height of 5 feet and displays deep red leaves. Great rose
        mallow (Hibiscus grandiflorus) is a perennial species that displays
        light pink blooms at a height of 8 feet, according to the Clemson
        University Extension. Additionally, hollyhocks (Alcea rosea) often reach
        8 feet in height and display flowers in vivid colors
      - >-
        Snake bites to people tend to be warning bites, and as such contain
        little venom. The most common venomous snake in the eastern states,
        copperheads are considered pit vipers, but unlike most other vipers, the
        copperhead does not flee when it is caught unawares. Instead, the snake
        will freeze in its current position. Of all the pit vipers, copperhead
        venom is the least toxic. Breeding does not take place every year, but a
        female snake will give birth to live young. Litters can consist of up to
        twenty young, though fewer than ten is most common. As with the majority
        of reptiles, the babies are on their own once they are born.
  - source_sentence: how long crocodile live without food
    sentences:
      - >-
        Copper is a chemical element with symbol Cu (from Latin: cuprum) and
        atomic number 29. It is a ductile metal with very high thermal and
        electrical conductivity. Pure copper is soft and malleable; a freshly
        exposed surface has a reddish-orange color. It is used as a conductor of
        heat and electricity, a building material, and a constituent of various
        metal alloys.
      - >-
        Watercress, a slightly sweet and spicy green that you won’t find at
        every market, is an amazingly delicious green to enjoy when you get the
        chance. Reminiscent of arugula and spinach combined, you’ll find it
        often still with the roots attached or even sold in small water pots at
        stores like Whole Foods. The cruciferous veggies like watercress, kale,
        broccoli, cabbage, etc., all topped the list while other leafy greens
        such as spinach, romaine, and beet greens also ranked high on the list.
      - >-
        Share to: about 1 week actually, but most people say 2 weeks, but that
        is a long time if you think about it. New answer; People who
        deliberatley stop eating can go for about 2 weeks, an … d it tends to be
        skinny people who do this. You can go a long time without food but not
        even 2 days or so without water....

SentenceTransformer based on distilbert/distilbert-base-uncased

This is a sentence-transformers model finetuned from distilbert/distilbert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DistilBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("aryanmagoon/ms_marco_bi_encoder")
# Run inference
sentences = [
    'how long crocodile live without food',
    'Share to: about 1 week actually, but most people say 2 weeks, but that is a long time if you think about it. New answer; People who deliberatley stop eating can go for about 2 weeks, an … d it tends to be skinny people who do this. You can go a long time without food but not even 2 days or so without water....',
    'Copper is a chemical element with symbol Cu (from Latin: cuprum) and atomic number 29. It is a ductile metal with very high thermal and electrical conductivity. Pure copper is soft and malleable; a freshly exposed surface has a reddish-orange color. It is used as a conductor of heat and electricity, a building material, and a constituent of various metal alloys.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 676,193 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 4 tokens mean: 9.15 tokens max: 23 tokens	min: 17 tokens mean: 96.96 tokens max: 254 tokens	min: 0.0 mean: 0.13 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  what airport is closest to rinteln germany	What is the closest airport to Berlin, Germany? The closest international and non-international airports to Berlin, Germany are listed below in order of increasing distance.	0.0
  what is javaone	JavaOne™ coffee pods are individually engineered with the precise roast level, grind setting, blending and dosage to achieve the best tasting pods. Starting with only the finest quality Arabica coffee beans, we roast our beans using hot air for a consistent, even roast throughout the entire bean. While traditional drum roasting can overcook the outside of the bean and undercook the inside, our beans are evenly roasted for a smoother, richer taste.	0.0
  what does watercress taste like	Watercress, a slightly sweet and spicy green that you won’t find at every market, is an amazingly delicious green to enjoy when you get the chance. Reminiscent of arugula and spinach combined, you’ll find it often still with the roots attached or even sold in small water pots at stores like Whole Foods. The cruciferous veggies like watercress, kale, broccoli, cabbage, etc., all topped the list while other leafy greens such as spinach, romaine, and beet greens also ranked high on the list.	0.0

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss
0.1893	500	1.2126
0.3786	1000	0.2246
0.5680	1500	0.1542
0.7573	2000	0.1332
0.9466	2500	0.115
1.1359	3000	0.1025
1.3253	3500	0.0929
1.5146	4000	0.081
1.7039	4500	0.074
1.8932	5000	0.0669
2.0825	5500	0.0605
2.2719	6000	0.0563
2.4612	6500	0.047
2.6505	7000	0.0433
2.8398	7500	0.0391

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 3.1.1
• Transformers: 4.45.1
• PyTorch: 2.4.1+cu121
• Accelerate: 0.34.2
• Datasets: 3.0.1
• Tokenizers: 0.20.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}