jaimevera1107/all-MiniLM-L6-v2-pubmed

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("jaimevera1107/all-MiniLM-L6-v2-pubmed")
# Run inference
sentences = [
    'The ionic currents in the tunicate egg Halocynthia roretzi were found to be solely dependent on potassium concentration, with no influence from sodium.',
    'The findings indicated that the ionic currents were completely abolished by the replacement of sodium with potassium ions, showing no other components.',
    'The cattle tick Hyalomma anatolicum exhibits varying radiation tolerance levels, with unfed adults tolerating up to 1000 R for engorgement and reproduction, while engorged females can tolerate 10,000 R for oviposition, but higher doses inhibit egg-laying.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 67,560 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 4 tokens mean: 73.08 tokens max: 256 tokens	min: 4 tokens mean: 61.28 tokens max: 256 tokens	min: 0.0 mean: 0.69 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Nonstimulated lymphocytes infected with measles virus do not express detectable virus antigens, but upon stimulation with phytohemagglutinin, the virus can be reactivated, leading to increased virus production and cell death.	Nonstimulated lymphocytes infected with measles virus produce detectable virus antigens immediately upon stimulation with phytohemagglutinin.	0.5
  In vitro studies of axillary lymph node cells in patients with breast cancer. A total of 170 axillary lymph nodes were obtained from fresh mastectomy specimens from 81 women with breast cancer. Lymph node cells were tested in vitro for T and B cells by the rosette technique and immunofluorescence microscopy and for functional capacity by response to the mitogens phytohemagglutinin (PHA) and concanavalin A. T cells showed a wide range of relative values: 32-80 percent, with a mean of 63.5 percent. B cells defined by the presence of surface immunoglobulins ranged from 14 to 61 percent (mean, 35.8 percent); those defined by the presence of C3 receptors, from 8 to 54 percent (mean, 24.9 percent); and those defined by the presence of IgG-specific (Fc) receptors, from 10 to 45 percent (mean, 27.5 percent). Cells with the C3 and Fc receptors constituted approximately two-thirds of the cells not binding spontaneously to sheep red blood cells (non-SRBC-R), whereas virtually all non-SRBC-R stain...	In a study of axillary lymph nodes from 81 breast cancer patients, T and B cell proportions varied significantly by age, metastatic status, and lymph node morphology, with older patients and nodes with metastasis showing higher B cell and lower T cell percentages.	1.0
  Pharmacologic treatment of disorders of bladder and urethra: a review. The use of pharmacologic agents in treating disorders of the bladder and proximal urethra has expanded because of new knowledge gained in the past few years. A better understanding of the properties of these organs as they relate to drugs has contributed to this expansion. The authors present their experience with a number of drugs in treating disorders of the detrusor muscle and proximal urethra, and they briefly review the literature.	Recent advancements in understanding bladder and urethra properties have led to an expanded use of pharmacologic agents for treating related disorders.	1.0

• Loss: CosineSimilarityLoss with these parameters:

  {
      "loss_fct": "torch.nn.modules.loss.MSELoss"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• num_train_epochs: 4
• fp16: True
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss
0.2367	500	0.065
0.4735	1000	0.041
0.7102	1500	0.0356
0.9470	2000	0.0319
1.1837	2500	0.0287
1.4205	3000	0.0272
1.6572	3500	0.0262
1.8939	4000	0.0263
2.1307	4500	0.0252
2.3674	5000	0.0228
2.6042	5500	0.0225
2.8409	6000	0.0221
3.0777	6500	0.0219
3.3144	7000	0.02
3.5511	7500	0.0198
3.7879	8000	0.0203

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 4.1.0
• Transformers: 4.52.3
• PyTorch: 2.7.0+cu118
• Accelerate: 1.7.0
• Datasets: 3.6.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}