SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/all-mpnet-base-v2
Maximum Sequence Length: 384 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '41 year old man with history of severe intellectual disability, CHF, epilepsy presenting with facial twitching on the right and generalized shaking in at his NH which required 20 mg valium to cease seizure activity. Per outside medical patient was felt to have focal epilepsy with secondary generalization, likely due to anoxic brain injury at birth, and probably related to the atrophic changes seen on MRI, particularly in the left temporal lobe.\nThe patient first developed seizures at age 13 found by family to have a generalized convulsion. He had a second seizure two years after his first episode. He was maintained on Dilantin and phenobarbital. The patient went 20 years without another seizure. He was recently tapered off Dilantin, and it was felt that perhaps this medication was necessary to maintain him seizure free. The patient had no further events during the hospital course and was back at his baseline at the time of discharge. Full EEG reports are pending at the time of dictation.\nPast Medical History:\nEpilepsy as above, CHF, depression',
    'Mesial temporal lobe epilepsy (MTLE) is the most common cause of medication-resistant\r\n      epilepsy in adults. The standard treatment for refractory MTLE is surgical resection by\r\n      craniotomy. Stereotactic laser interstitial thermal therapy (LITT) is a new surgical\r\n      technique being used to treat MTLE. Under MRI-guidance, a laser probe is inserted into the\r\n      seizure focus and heat is used to destroy the tissue. Compared to temporal lobectomy, LITT\r\n      results in shorter hospital stays, low complication rates, and possibly less cognitive\r\n      decline; however, seizure freedom rates are potentially lower.\r\n\r\n      During temporal lobectomy, neurophysiologic intraoperative monitoring (NIOM) can be used to\r\n      better identify epileptogenic tissue and guide resection. This tool has been unavailable\r\n      during LITT procedures. Recently, the investigators demonstrated in two cases that NIOM with\r\n      a depth electrode is technically feasible during LITT and can identify epileptiform activity\r\n      intra-operatively.\r\n\r\n      This is a prospective trial of NIOM during LITT for mesial temporal lobe epilepsy. The\r\n      investigators will assess the safety of performing NIOM during LITT and whether data from\r\n      NIOM (frequency and characteristics of epileptiform discharges recorded before and after\r\n      ablation) are associated with seizure outcomes. If there is an association, NIOM could be\r\n      used for prognostication and could potentially even be used to guide surgery.\r\n\r\n      Hypotheses:\r\n\r\n        1. NIOM performed by MRI-guided stereotactic depth electrode placed in the parahippocampal\r\n           gyrus adjacent to the LITT catheter is safe, as compared to institutional LITT controls\r\n           without NIOM and published LITT complication rates.\r\n\r\n        2. Greater magnitude fractional decrements in discharge frequency from pre-ablation to\r\n           post-ablation recordings will be significantly associated with better seizure outcomes,\r\n           as measured by International League Against Epilepsy (ILAE) surgical outcome scores.\r\n\r\n      Objectives:\r\n\r\n      The primary project goals are to assess if NIOM by parahippocampal depth electrode is safe\r\n      during LITT of MTLE and to assess if the fractional decrement of interictal discharges (ID)\r\n      on NIOM can be significantly correlated with outcome. The study will be powered to address\r\n      these questions a priori. Post hoc analyses consisting of a multivariate analysis of other\r\n      patient demographic data, NIOM findings, operative parameters, quality of life scores, and\r\n      neuropsychiatric outcomes will also be assessed.',
    "BACKGROUND There is no consensus regarding the injury mechanism in complex prolonged Whiplash\r\n      Associated Disorders (WAD) cases. Often, tissue damage and physiological alterations is not\r\n      detectable. In order to improve future rehabilitation, a greater understanding of the\r\n      mechanisms underlying whiplash injury and their importance for treatment success is required.\r\n      It is also important to investigate if pathophysiological changes can be restored by\r\n      rehabilitation.\r\n\r\n      AIM The projects aims to investigate neck muscle structure and function, biomarkers and the\r\n      association with pain, disability and other outcomes before and after neck-specific\r\n      exercises.\r\n\r\n      METHODS Design These are sub-group trials, each one independent of the others, in a\r\n      prospective, multicentre, randomized controlled trial (RCT) with two parallel treatment arms\r\n      conducted according to a protocol established before recruitment started (ClinicalTrials.gov\r\n      Protocol ID: NCT03022812). Physiotherapist-led neck-specific exercise previously shown to be\r\n      effective for the current population constitutes the control treatment for the new\r\n      Internet-based neck-specific exercise treatment. In the RCT, a total of 140 patients will be\r\n      included (70 from each group), whereof 30 (both randomization arms equally) consecutively\r\n      will be asked to participate in the present sub-group study. The sub-group studies aims to\r\n      start September 2019. Independent physiotherapists in primary health care will distribute the\r\n      treatment.\r\n\r\n      In sub-group of individuals, additional measurements will be performed before and after\r\n      interventions end (3 months follow-up). The physical measurements will be performed by\r\n      independent specially trained test-leaders blinded to randomization.\r\n\r\n      Additionally, 30 neck healthy individuals without serious diseases matched for age and gender\r\n      will consecutively be investigated.\r\n\r\n      Study population\r\n\r\n      The inclusion criteria for patients are:\r\n\r\n        -  Chronic neck problems corresponding to WAD grades 2-3 verified by clinical examination\r\n\r\n        -  Average estimated pain in the last week at least 20 mm on the visual analogue scale\r\n           (VAS)\r\n\r\n        -  Neck disability of more than 20% on the Neck Disability Index (NDI) [10]\r\n\r\n        -  Working age (18 - 63 years)\r\n\r\n        -  Daily access to a computer/tablet/smart phone and Internet\r\n\r\n        -  Neck symptoms within the first week after the injury (i.e., neck pain, neck stiffness,\r\n           or cervical radiculopathy).\r\n\r\n      For the present sub group study additional criteria were:\r\n\r\n        -  Right handed\r\n\r\n        -  Dominant right sided or equal sided pain\r\n\r\n      Inclusion criteria for healthy controls:\r\n\r\n      • Age and gender matched healthy individuals without neck pain and disability (VAS <10mm, NDI\r\n      <5%) that feel overall healthy without known diseases.\r\n\r\n      Exclusion criteria for patients:\r\n\r\n        -  Individuals with any of the following signs of head injury at the time of whiplash\r\n           injury will be excluded: loss of consciousness, amnesia before or after the injury,\r\n           altered mental status (e.g., confusion, disorientation), focal neurological changes\r\n           (changes in smell and taste).\r\n\r\n        -  Previous fractures or dislocation of the cervical spine\r\n\r\n        -  Known or suspected serious physical pathology included myelopathy,\r\n\r\n        -  Spinal tumours\r\n\r\n        -  Spinal infection\r\n\r\n        -  Ongoing malignancy\r\n\r\n        -  Previous severe neck problems that resulted in sick leave for more than a month in the\r\n           year before the current whiplash injury\r\n\r\n        -  surgery in the cervical spine\r\n\r\n        -  Generalized or more dominant pain elsewhere in the body\r\n\r\n        -  Other illness/injury that may prevent full participation\r\n\r\n        -  Inability to understand and write in Swedish\r\n\r\n      Additional criteria in the present sub group:\r\n\r\n        -  Increased risk of bleeding,\r\n\r\n        -  BMI >35\r\n\r\n        -  Contraindications of MRI such as metal, severe obesity, pacemaker and pregnancy.\r\n\r\n      Exclusion criteria for healthy controls:\r\n\r\n        -  Earlier neck injury,\r\n\r\n        -  Recurrent neck pain,\r\n\r\n        -  Earlier treatment for neck pain.\r\n\r\n        -  Increased risk of bleeding,\r\n\r\n        -  BMI >35\r\n\r\n        -  Contraindications of MRI\r\n\r\n      Recruitment and randomization Information about the study will be provided by healthcare\r\n      providers, reports in newspapers, social media, and the university's website. Interested\r\n      patients will contact the research team through the project website. After completing a small\r\n      survey on the website, a project team member (physiotherapist) will perform a telephone\r\n      interview and ask about the patient's medical history. An appointment for a physical\r\n      examination and additional interview for the present sub-group study is made as a last step\r\n      to ensure that the criteria for study participation are met. If the study criteria are met,\r\n      written and oral informed consent are obtained, and the patient will fill out a questionnaire\r\n      and undergo physical measurements of neck-related function. Baseline measurements must be\r\n      completed for inclusion.\r\n\r\n      Healthy individuals will consecutively be recruited among friends, family and staff at the\r\n      university or the university hospital to suit the age and gender of a patient.\r\n\r\n      Intervention for the patient group The intervention consists of neck-specific exercises\r\n      distributed in two different ways, twice a week at the physiotherapist clinic for 3 months\r\n      (NSE group) or with 4 physiotherapy visits only combined with a web-based system (NSEIT\r\n      group).\r\n\r\n      A. In the NSE group, patients will get an explanation and justification for the exercise\r\n      consisting of basic information about the musculoskeletal anatomy of the neck relevant to the\r\n      exercises given by the physiotherapist in order to motivate the patient and help make them\r\n      feel safe and reassured. The patients undergo a 12-week training programme with a\r\n      physiotherapist 2 days/week (total 24 times). Exercises are chosen from a clear and written\r\n      frame of exercises. The training includes exercises for the deep neck muscles, continuing\r\n      with the endurance training of neck and shoulder muscles. The exercises are individually\r\n      adjusted according to the individual's physical conditions and progressively increased in\r\n      severity and dose. Exercise-related pain provocation is not accepted. The patient may also\r\n      perform exercises at home. At the end of the treatment period, the participants are\r\n      encouraged to continue practising on their own. The exercises have been used with good\r\n      results in previous RCTs.\r\n\r\n      B. In the NSEIT group, patients will receive the same information and training programmes as\r\n      the NSE group, but with 4 visits to the physiotherapist instead of 24. Exercises are\r\n      introduced, progressed, and followed up to ensure correct performance. The exercises are\r\n      performed and most of the information is given with the help of Internet support outside the\r\n      healthcare system. Photos and videos of the exercises (a clear stepwise progression) and\r\n      information are available on the Web-based system. A SMS reminder is automatically available\r\n      if the exercise diary is not completed. The time required for training is the same as in\r\n      group A, but without the patient having to go to the physiotherapy clinic. The Internet\r\n      programme was developed by experienced physiotherapists/ researchers together with\r\n      technicians and clinicians. Technicians are available to assist the participants if any\r\n      technical difficulties arise. The patients will be introduced to the exercises and get\r\n      information and support at the physiotherapy visits.\r\n\r\n      Variables and measurements Background data and data in the RCT include personal details,\r\n      questionnaires and test results regarding pain, physical and psychological functioning,\r\n      health and cost-effectiveness described elsewhere (ClinicalTrials.gov Protocol ID:\r\n      NCT03022812).\r\n\r\n      Measurements will be done at baseline for both groups and at 3 months follow-up for the\r\n      patient when treatment ends. Except for blood and saliva samples that will be collected twice\r\n      (baseline and repeated after 3 months), the measurements will be performed at baseline only\r\n      for the healthy individuals.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 11,583 training samples
Columns: sentence_0, sentence_1, and sentence_2

Approximate statistics based on the first 1000 samples:

	sentence_0	sentence_1	sentence_2
type	string	string	string
details	min: 82 tokens mean: 214.67 tokens max: 355 tokens	min: 4 tokens mean: 227.41 tokens max: 384 tokens	min: 4 tokens mean: 259.12 tokens max: 384 tokens

Samples:

sentence_0	sentence_1	sentence_2
The patient is a 35-year-old woman with myasthenia gravis, class IIa. She complains of diplopia and fatigue and weakness that affects mainly her upper limbs. She had a positive anti-AChR antibody test, and her single fiber electromyography (SFEMG) was positive. She takes pyridostigmine 60 mg three times a day. But she still has some symptoms that interfere with her job. She is a research coordinator and has 3 children. Her 70-year-old father has hypertension. She does not smoke or use illicit drugs. She drinks alcohol occasionally at social events. Her physical exam and lab studies were not remarkable for any other abnormalities. BP: 110/75 Hgb: 11 g/dl WBC: 8000 /mm3 Plt: 300000 /ml Creatinine: 0.5 mg/dl BUN: 10 mg/dl Beta hcg: negative for pregnancy	`Randomized, double-blind, placebo-controlled, parallel group study is designed to evaluate`
the safety, tolerability and efficacy of amifampridine phosphate in patients with MuSK-MG. In
addition, a sample of AChR-MG patients will be assess for efficacy and safety of
amifampridine phosphate. Planned duration of participation for each patient is at least 38
days, excluding the screening period. Eligible patients will be titrated to an efficacious
dose of amifampridine phosphate and those who demonstrate improvement will be randomized to
either placebo or amifampridine, in a double-blind fashion, for 10 days.	`this randomized controlled trial will compare the impact of routine use of completion`
angiography versus using it on selective bases after thromboembolectomy in patients with
acute lower limb ischemia and their impact on limb salvage rates
The patient is a 17-year-old boy complaining of severe migratory pain in the right lower quadrant of his abdomen that started four days ago. The pain is accompanied by nausea and vomiting. He was febrile with tenderness, rebound tenderness and guarding on palpation. His WBC was elevated with dominant neutrophils. CT scan showed evidence of acute perforated appendicitis with free fluid in the pelvis. Diagnostic laparoscopy revealed phlegmon with no other abdominal abnormalities. He is now a candidate for emergent laparoscopic appendectomy under general anesthesia.	`Acute appendicitis is one of the most common causes of abdominal pain in emergency`
departments as well as one of the most common indications for emergency abdominal surgery.
The clinical diagnosis has been based on patient history, physical examination and laboratory
findings as well as the "clinical eye" of the surgeon. Still the diagnosis remains
challenging. One of the main problems is that many other disorders can mimic the clinical
presentation of appendicitis, thus increasing the role of imaging techniques to aid in
diagnostic accuracy. Now preoperative imaging in patients with suspected acute appendicitis
is currently widely accepted as the standard of practice, and CT has been shown to clearly
outperform US in terms of diagnostic performance. Nowadays, CT imaging is considered the
primary imaging modality in the diagnosis for acute appendicitis as it is appraised for its
high sensitivity and specificity. The ...	`Urticaria is a common skin disorder that is classified according to its chronicity into acute`
and chronic forms. It may occur spontaneously or on exposure to a physical factor. In the
latter case, the urticaria is classified as a physical urticaria . Physical urticaria may be
induced by mechanical and applied pressure, exercise, or exposure to cold, heat, sun, water,
or vibration. The urticarial lesions are generally thought to be the result of mast cell
activation and degranulation, which is supported by the finding of increased levels of serum
histamine during some urticarial flares. Passive transfer experiments, whereupon serum from
affected donors is transferred into recipient s skin followed by physical stimulation with
resultant urticaria at the site of challenge, have been positive in some instances. This
suggests the presence of an intrinsic factor in serum, such as IgE, which then mediates
activation of tiss...
34 year old woman with Marfan's syndrome and known severe mitral valve prolapse with regurgitation, who was planned for a MV repair but was lost to follow-up. She remains symptomatic and is now prepared to undergo mitral valve repair/replacement surgery. EF of 65% on TTE. Past Medical History: Marfans Syndrome MVP with severe mitral regurgitation Gastric reflux disease History of gestational diabetes mellitus Hypertension with pregnancy Obesity c-section x 2 laser eye surgery cataract surgery foot surgery (shorten bone length)	`Early feasibility study - multi-center, prospective, single-arm, and non-randomized study`
without concurrent or historical controls.

The primary objective of the study is to generate early feasibility data for the CardiAQ™
Transcatheter Mitral Valve Implant System with the Transfemoral and Transapical Delivery
Systems for the treatment of moderate to severe mitral valve regurgitation in patients who
are considered high risk for mortality and morbidity from conventional open-heart surgery.

The secondary objectives of the study are to evaluate the long-term safety of the device and
the effects of the device on performance, functional, quality of life parameters, and
technical, device, procedural, and individual patient successes.

The study is to be performed at a maximum of 5 investigational sites in the US.	`Acute kidney injury (AKI) is a common complication in patients suffering from acute coronary`
syndromes (ACS) and treated by percutaneous coronary intervention (PCI). This complication
has been associated with higher early and late adverse events. It has been emphasized that
the pathogenesis of AKI in the setting of ACS is multifactorial, including age, unstable
hemodynamic conditions, co-morbidities (that is, diabetes mellitus and anemia) pre-existing
chronic kidney disease, dehydration and administration of nephrotoxic drugs. However, the
role of iodinated contrast media (CM) has been well established. Hydration represents the
cornerstone in contrast-induced AKI (CI-AKI) prevention. However, at present there is no
consensus on how hydration should be carried out, especially in ACS patients, and all the the
recommended hydration regimens have limited applicability in the urgent/emergent settings
such as ACS. Several ...

Loss: TripletLoss with these parameters:

{
    "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
    "triplet_margin": 5
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 2
per_device_eval_batch_size: 2
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 2
per_device_eval_batch_size: 2
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 3
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
tp_size: 0
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss
0.0863	500	4.6761
0.1727	1000	4.4618
0.2590	1500	4.3825
0.3453	2000	4.3709
0.4316	2500	4.2951
0.5180	3000	4.322
0.6043	3500	4.2719
0.6906	4000	4.2655
0.7769	4500	4.2715
0.8633	5000	4.2587
0.9496	5500	4.169
1.0359	6000	4.1168
1.1222	6500	4.0476
1.2086	7000	4.0758
1.2949	7500	4.0531
1.3812	8000	4.0327
1.4675	8500	4.0836
1.5539	9000	4.1076
1.6402	9500	4.0086
1.7265	10000	4.0768
1.8128	10500	4.0136
1.8992	11000	3.9689
1.9855	11500	4.059
2.0718	12000	3.9517
2.1581	12500	3.9293
2.2445	13000	3.9178
2.3308	13500	3.98
2.4171	14000	3.9394
2.5035	14500	3.9541
2.5898	15000	3.8973
2.6761	15500	3.9268
2.7624	16000	3.8798
2.8488	16500	3.8903
2.9351	17000	3.939

Framework Versions

Python: 3.11.11
Sentence Transformers: 3.4.1
Transformers: 4.50.2
PyTorch: 2.6.0+cu124
Accelerate: 1.5.2
Datasets: 3.5.0
Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}