pavanmantha/embeddinggemma-pubmed

SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m on the pubmed-30k dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m
• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • pubmed-30k

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pavanmantha/embeddinggemma-pubmed")
# Run inference
queries = [
    "Can you deliver accurate tidal volume by manual resuscitator?",
]
documents = [
    'One of the problems with manual resuscitators is the difficulty in achieving accurate volume delivery. The volume delivered to the patient varies by the physical characteristics of the person and method. This study was designed to compare tidal volumes delivered by the squeezing method, physical characteristics and education and practice levels.',
    'Although current resuscitation guidelines are rescuer focused, the opportunity exists to develop patient-centered resuscitation strategies that optimize the hemodynamic response of the individual in the hopes to improve survival.',
    'Sections from paraffin-embedded blocks of surgically resected specimens of GBC (69 cases), XGC (65), chronic cholecystitis (18) and control gallbladder (10) were stained with the monoclonal antibodies to p53 and PCNA, and a polyclonal antibody to beta-catenin. p53 expression was scored as the percentage of nuclei stained. PCNA expression was scored as the product of the percentage of nuclei stained and the intensity of the staining (1-3). A cut-off value of 80 for this score was taken as a positive result. Beta-catenin expression was scored as type of expression-membranous, cytoplasmic or nuclear staining.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.9156,  0.2237, -0.1894]])

Training Details

Training Dataset

pubmed-30k

• Dataset: pubmed-30k at 6a7c15c
• Size: 33,200 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 11 tokens mean: 17.74 tokens max: 36 tokens	min: 4 tokens mean: 85.76 tokens max: 301 tokens	min: 5 tokens mean: 82.14 tokens max: 409 tokens

• Samples:

  anchor	positive	negative
  Does a history of unintended pregnancy lessen the likelihood of desire for sterilization reversal?	Unintended pregnancy has been significantly associated with subsequent female sterilization. Whether women who are sterilized after experiencing an unintended pregnancy are less likely to express desire for sterilization reversal is unknown.	Changes in serum hormone levels induced by combined contraceptives.
  Does a history of unintended pregnancy lessen the likelihood of desire for sterilization reversal?	Unintended pregnancy has been significantly associated with subsequent female sterilization. Whether women who are sterilized after experiencing an unintended pregnancy are less likely to express desire for sterilization reversal is unknown.	Evolutionary life history theory predicts that, in the absence of contraception, any enhancement of maternal condition can increase human fertility. Energetic trade-offs are likely to be resolved in favour of maximizing reproductive success rather than health or longevity. Here we find support for the hypothesis that development initiatives designed to improve maternal and child welfare may also incur costs associated with increased family sizes if they do not include a family planning component.
  Does a history of unintended pregnancy lessen the likelihood of desire for sterilization reversal?	Unintended pregnancy has been significantly associated with subsequent female sterilization. Whether women who are sterilized after experiencing an unintended pregnancy are less likely to express desire for sterilization reversal is unknown.	Out of 663 cycles resulting in oocyte retrieval, 299 produced a clinical pregnancy (45.1%). Women who achieved a clinical pregnancy had a significantly shorter stimulation length (11.9 vs. 12.1 days, p = 0.047). Polycystic ovary syndrome (PCOS) was the only etiology of infertility that was significantly associated with a higher chance for clinical pregnancy and was a significant confounder for the association of duration and success of treatment. Women with 13 days or longer of stimulation had a 34 % lower chance of clinical pregnancy as compared to those who had a shorter cycle (OR 0.66, 95% CI:0.46-0.95) after adjustment for age, ovarian reserve, number of oocytes retrieved, embryos transferred and PCOS diagnosis.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• learning_rate: 2e-05
• warmup_steps: 0.1
• gradient_accumulation_steps: 4
• fp16: True
• warmup_ratio: 0.1
• prompts: task: sentence similarity | query:

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 4
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: no
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: 0.1
• local_rank: -1
• prompts: task: sentence similarity | query:
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.1928	100	0.2086
0.3855	200	0.0872
0.5783	300	0.0623
0.7711	400	0.0569
0.9639	500	0.0487
1.1561	600	0.0423
1.3489	700	0.0412
1.5417	800	0.0407
1.7345	900	0.0341
1.9272	1000	0.0384
2.1195	1100	0.0316
2.3123	1200	0.0290
2.5051	1300	0.0314
2.6978	1400	0.0303
2.8906	1500	0.0245

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 5.2.3
• Transformers: 5.2.0
• PyTorch: 2.8.0.dev20250319+cu128
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}