--- tags: - sentence-transformers - sentence-similarity - feature-extraction - dense - generated_from_trainer - dataset_size:33200 - loss:MultipleNegativesRankingLoss base_model: google/embeddinggemma-300m widget: - source_sentence: Are stroke patients' reports of home blood pressure readings reliable? sentences: - The first Whitehall study. - A total of 1027 monitor and 716 booklet readings were recorded. Ninety per cent of booklet recordings were exactly the same as the BP monitor readings. Average booklet readings were 0.6 mmHg systolic [95% confidence interval (95% CI) -0.6 to 1.8] and 0.3 mmHg diastolic (95% CI -0.3 to 0.8) lower than those on the monitor. - 'Protocol 1: a) office blood pressure measurement and Home1 were significantly higher than ambulatory blood pressure monitoring, except for systolic and diastolic office blood pressure measurement taken by the patient or a family member, systolic blood pressure taken by a nurse, and diastolic blood pressure taken by a physician. b) ambulatory blood pressure monitoring and HBPM1 were similar. Protocol 2: a) HBPM2 and Home2 were similar. b) Home2 was significantly lower than Home1, except for diastolic blood pressure taken by a nurse or the patient. There were significant relationships between: a) diastolic blood pressure measured by the patient and the thickness of the interventricular septum, posterior wall, and left ventricular mass; and b) ambulatory and HBPM2 diastolic and systolic blood pressure taken by a physician (home2) and left ventricular mass. Therefore, the data indicate that home blood pressure measurement and ambulatory blood pressure monitoring had good prognostic values relative to "office measurement."' - source_sentence: Do socioeconomic differences in mortality persist after retirement? sentences: - to compare the mortality rates of elderly demented and nondemented subjects and the differential association of midlife risk factors with mortality according to dementia status. - Death. - To investigate polysomnographic and anthropomorphic factors predicting need of high optimal continuous positive airway pressure (CPAP). - source_sentence: Does a history of unintended pregnancy lessen the likelihood of desire for sterilization reversal? sentences: - Evolutionary life history theory predicts that, in the absence of contraception, any enhancement of maternal condition can increase human fertility. Energetic trade-offs are likely to be resolved in favour of maximizing reproductive success rather than health or longevity. Here we find support for the hypothesis that development initiatives designed to improve maternal and child welfare may also incur costs associated with increased family sizes if they do not include a family planning component. - This study used national, cross-sectional data collected by the 2006-2010 National Survey of Family Growth. The study sample included women ages 15-44 who were surgically sterile from a tubal sterilization at the time of interview. Multivariable logistic regression was used to examine the relationship between a history of unintended pregnancy and desire for sterilization reversal while controlling for potential confounders. - Anti-HTLV-I antibodies were positive in both the serum and the CSF in all of the patients. Biopsied sample from spinal cord lesions showed inflammatory changes in Patient 1. Patient 2 had a demyelinating type of sensorimotor polyneuropathy. Two of the three patients examined showed high risk of developing HAM/TSP in virologic and immunologic aspects. - source_sentence: Are behavioural risk factors to be blamed for the conversion from optimal blood pressure to hypertensive status in Black South Africans? sentences: - Longitudinal cohort studies in sub-Saharan Africa are urgently needed to understand cardiovascular disease development. We, therefore, explored health behaviours and conventional risk factors of African individuals with optimal blood pressure (BP) (≤ 120/80 mm Hg), and their 5-year prediction for the development of hypertension. - The primary aim was to assess long-term blood pressure in 110 patients with Type 2 diabetes who had achieved optimal blood pressure control during attendance at a protocol-based nurse-led hypertension intensive intervention clinic 7 years previously. The secondary aim was to assess modifiable cardiovascular risk factor status. - The Prospective Urban Rural Epidemiology study in the North West Province, South Africa, started in 2005 and included African volunteers (n = 1994; aged>30 years) from a sample of 6000 randomly selected households in rural and urban areas. - source_sentence: Can you deliver accurate tidal volume by manual resuscitator? sentences: - One of the problems with manual resuscitators is the difficulty in achieving accurate volume delivery. The volume delivered to the patient varies by the physical characteristics of the person and method. This study was designed to compare tidal volumes delivered by the squeezing method, physical characteristics and education and practice levels. - Sections from paraffin-embedded blocks of surgically resected specimens of GBC (69 cases), XGC (65), chronic cholecystitis (18) and control gallbladder (10) were stained with the monoclonal antibodies to p53 and PCNA, and a polyclonal antibody to beta-catenin. p53 expression was scored as the percentage of nuclei stained. PCNA expression was scored as the product of the percentage of nuclei stained and the intensity of the staining (1-3). A cut-off value of 80 for this score was taken as a positive result. Beta-catenin expression was scored as type of expression-membranous, cytoplasmic or nuclear staining. - Although current resuscitation guidelines are rescuer focused, the opportunity exists to develop patient-centered resuscitation strategies that optimize the hemodynamic response of the individual in the hopes to improve survival. datasets: - pavanmantha/pubmed-30k pipeline_tag: sentence-similarity library_name: sentence-transformers --- # SentenceTransformer based on google/embeddinggemma-300m This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [google/embeddinggemma-300m](https://huggingface.co/google/embeddinggemma-300m) on the [pubmed-30k](https://huggingface.co/datasets/pavanmantha/pubmed-30k) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [google/embeddinggemma-300m](https://huggingface.co/google/embeddinggemma-300m) - **Maximum Sequence Length:** 2048 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - [pubmed-30k](https://huggingface.co/datasets/pavanmantha/pubmed-30k) ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'}) (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'}) (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'}) (4): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("pavanmantha/embeddinggemma-pubmed") # Run inference queries = [ "Can you deliver accurate tidal volume by manual resuscitator?", ] documents = [ 'One of the problems with manual resuscitators is the difficulty in achieving accurate volume delivery. The volume delivered to the patient varies by the physical characteristics of the person and method. This study was designed to compare tidal volumes delivered by the squeezing method, physical characteristics and education and practice levels.', 'Although current resuscitation guidelines are rescuer focused, the opportunity exists to develop patient-centered resuscitation strategies that optimize the hemodynamic response of the individual in the hopes to improve survival.', 'Sections from paraffin-embedded blocks of surgically resected specimens of GBC (69 cases), XGC (65), chronic cholecystitis (18) and control gallbladder (10) were stained with the monoclonal antibodies to p53 and PCNA, and a polyclonal antibody to beta-catenin. p53 expression was scored as the percentage of nuclei stained. PCNA expression was scored as the product of the percentage of nuclei stained and the intensity of the staining (1-3). A cut-off value of 80 for this score was taken as a positive result. Beta-catenin expression was scored as type of expression-membranous, cytoplasmic or nuclear staining.', ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) print(query_embeddings.shape, document_embeddings.shape) # [1, 768] [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) # tensor([[ 0.9156, 0.2237, -0.1894]]) ``` ## Training Details ### Training Dataset #### pubmed-30k * Dataset: [pubmed-30k](https://huggingface.co/datasets/pavanmantha/pubmed-30k) at [6a7c15c](https://huggingface.co/datasets/pavanmantha/pubmed-30k/tree/6a7c15c83164ef44a767f4da72b5e71bd920104f) * Size: 33,200 training samples * Columns: anchor, positive, and negative * Approximate statistics based on the first 1000 samples: | | anchor | positive | negative | |:--------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | anchor | positive | negative | |:----------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Does a history of unintended pregnancy lessen the likelihood of desire for sterilization reversal? | Unintended pregnancy has been significantly associated with subsequent female sterilization. Whether women who are sterilized after experiencing an unintended pregnancy are less likely to express desire for sterilization reversal is unknown. | Changes in serum hormone levels induced by combined contraceptives. | | Does a history of unintended pregnancy lessen the likelihood of desire for sterilization reversal? | Unintended pregnancy has been significantly associated with subsequent female sterilization. Whether women who are sterilized after experiencing an unintended pregnancy are less likely to express desire for sterilization reversal is unknown. | Evolutionary life history theory predicts that, in the absence of contraception, any enhancement of maternal condition can increase human fertility. Energetic trade-offs are likely to be resolved in favour of maximizing reproductive success rather than health or longevity. Here we find support for the hypothesis that development initiatives designed to improve maternal and child welfare may also incur costs associated with increased family sizes if they do not include a family planning component. | | Does a history of unintended pregnancy lessen the likelihood of desire for sterilization reversal? | Unintended pregnancy has been significantly associated with subsequent female sterilization. Whether women who are sterilized after experiencing an unintended pregnancy are less likely to express desire for sterilization reversal is unknown. | Out of 663 cycles resulting in oocyte retrieval, 299 produced a clinical pregnancy (45.1%). Women who achieved a clinical pregnancy had a significantly shorter stimulation length (11.9 vs. 12.1 days, p = 0.047). Polycystic ovary syndrome (PCOS) was the only etiology of infertility that was significantly associated with a higher chance for clinical pregnancy and was a significant confounder for the association of duration and success of treatment. Women with 13 days or longer of stimulation had a 34 % lower chance of clinical pregnancy as compared to those who had a shorter cycle (OR 0.66, 95% CI:0.46-0.95) after adjustment for age, ovarian reserve, number of oocytes retrieved, embryos transferred and PCOS diagnosis. | * Loss: [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim", "gather_across_devices": false } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 16 - `learning_rate`: 2e-05 - `warmup_steps`: 0.1 - `gradient_accumulation_steps`: 4 - `fp16`: True - `warmup_ratio`: 0.1 - `prompts`: task: sentence similarity | query: #### All Hyperparameters
Click to expand - `per_device_train_batch_size`: 16 - `num_train_epochs`: 3 - `max_steps`: -1 - `learning_rate`: 2e-05 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: None - `warmup_steps`: 0.1 - `optim`: adamw_torch_fused - `optim_args`: None - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `optim_target_modules`: None - `gradient_accumulation_steps`: 4 - `average_tokens_across_devices`: True - `max_grad_norm`: 1.0 - `label_smoothing_factor`: 0.0 - `bf16`: False - `fp16`: True - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `use_liger_kernel`: False - `liger_kernel_config`: None - `use_cache`: False - `neftune_noise_alpha`: None - `torch_empty_cache_steps`: None - `auto_find_batch_size`: False - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `include_num_input_tokens_seen`: no - `log_level`: passive - `log_level_replica`: warning - `disable_tqdm`: False - `project`: huggingface - `trackio_space_id`: trackio - `eval_strategy`: no - `per_device_eval_batch_size`: 8 - `prediction_loss_only`: True - `eval_on_start`: False - `eval_do_concat_batches`: True - `eval_use_gather_object`: False - `eval_accumulation_steps`: None - `include_for_metrics`: [] - `batch_eval_metrics`: False - `save_only_model`: False - `save_on_each_node`: False - `enable_jit_checkpoint`: False - `push_to_hub`: False - `hub_private_repo`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_always_push`: False - `hub_revision`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `restore_callback_states_from_checkpoint`: False - `full_determinism`: False - `seed`: 42 - `data_seed`: None - `use_cpu`: False - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `dataloader_prefetch_factor`: None - `remove_unused_columns`: True - `label_names`: None - `train_sampling_strategy`: random - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `ddp_backend`: None - `ddp_timeout`: 1800 - `fsdp`: [] - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `deepspeed`: None - `debug`: [] - `skip_memory_metrics`: True - `do_predict`: False - `resume_from_checkpoint`: None - `warmup_ratio`: 0.1 - `local_rank`: -1 - `prompts`: task: sentence similarity | query: - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | |:------:|:----:|:-------------:| | 0.1928 | 100 | 0.2086 | | 0.3855 | 200 | 0.0872 | | 0.5783 | 300 | 0.0623 | | 0.7711 | 400 | 0.0569 | | 0.9639 | 500 | 0.0487 | | 1.1561 | 600 | 0.0423 | | 1.3489 | 700 | 0.0412 | | 1.5417 | 800 | 0.0407 | | 1.7345 | 900 | 0.0341 | | 1.9272 | 1000 | 0.0384 | | 2.1195 | 1100 | 0.0316 | | 2.3123 | 1200 | 0.0290 | | 2.5051 | 1300 | 0.0314 | | 2.6978 | 1400 | 0.0303 | | 2.8906 | 1500 | 0.0245 | ### Framework Versions - Python: 3.11.11 - Sentence Transformers: 5.2.3 - Transformers: 5.2.0 - PyTorch: 2.8.0.dev20250319+cu128 - Accelerate: 1.12.0 - Datasets: 4.5.0 - Tokenizers: 0.22.2 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```