context_fine-tuned-SapBERT / README.md

Stevenf232

Add new SentenceTransformer model

1396374 verified 1 day ago

preview code

raw

history blame contribute delete

15.6 kB

metadata

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:27120
  - loss:ContrastiveLoss
base_model: cambridgeltl/SapBERT-from-PubMedBERT-fulltext
widget:
  - source_sentence: >-
      anencephaly [SEP] Sequential observations of exencephaly and subsequent
      morphological changes by mouse exo utero development system: analysis of t
    sentences:
      - >-
        Hemostatic Disorders [SEP] Pathological processes involving the
        integrity of blood circulation. Hemostasis depends on the integrity of
        BLOOD VESSELS, blood
      - >-
        Pentylenetetrazole [SEP] A pharmaceutical agent that displays activity
        as a central nervous system and respiratory stimulant. It is considered
        a non-comp
      - >-
        Epilepsy [SEP] A disorder characterized by recurrent episodes of
        paroxysmal brain dysfunction due to a sudden, disorderly, and excessive
        neuron
  - source_sentence: >-
      nifedipine [SEP] The effect of nifedipine on renal function in liver
      transplant recipients who were receiving tacrolimus was evaluated between
      Ja
    sentences:
      - >
        Nifedipine [SEP] A potent vasodilator agent with calcium antagonistic
        action. It is a useful anti-anginal agent that also lowers blood
        pressure.
      - >-
        Hypotension [SEP] Abnormally low BLOOD PRESSURE that can result in
        inadequate blood flow to the brain and other vital organs. Common
        symptom is DI
      - >-
        Granulomatosis with Polyangiitis [SEP] A multisystemic disease of a
        complex genetic background. It is characterized by inflammation of the
        blood vessels (VASCULITIS) l
  - source_sentence: >-
      toxicity [SEP] Effects of calcium channel blockers on bupivacaine-induced
      toxicity.
    sentences:
      - >-
        Methamphetamine [SEP] A central nervous system stimulant and
        sympathomimetic with actions and uses similar to DEXTROAMPHETAMINE. The
        smokable form is 
      - >-
        Dizocilpine Maleate [SEP] A potent noncompetitive antagonist of the NMDA
        receptor (RECEPTORS, N-METHYL-D-ASPARTATE) used mainly as a research
        tool. The dr
      - >-
        Hallucinations [SEP] Subjectively experienced sensations in the absence
        of an appropriate stimulus, but which are regarded by the individual as
        real.
  - source_sentence: >-
      Ca [SEP] Interactive effects of variations in [Na]o and [Ca]o on rat
      atrial spontaneous frequency.
    sentences:
      - >-
        Brain Edema [SEP] Increased intracellular or extracellular fluid in
        brain tissue. Cytotoxic brain edema (swelling due to increased
        intracellular f
      - >-
        Capsaicin [SEP] An alkylamide found in CAPSICUM that acts at TRPV CATION
        CHANNELS.
            
      - |-
        Thrombocytopenia [SEP] A subnormal level of BLOOD PLATELETS.
            
  - source_sentence: >-
      acromegaly [SEP] This article reports the changes in gallbladder function
      examined by ultrasonography in 20 Chinese patients with active acromega
    sentences:
      - >-
        Indocyanine Green [SEP] A tricarbocyanine dye that is used
        diagnostically in liver function tests and to determine blood volume and
        cardiac output.
            
      - >-
        Nitroglycerin [SEP] A volatile vasodilator which relieves ANGINA
        PECTORIS by stimulating GUANYLATE CYCLASE and lowering cytosolic
        calcium. It is als
      - >-
        Nausea [SEP] An unpleasant sensation in the stomach usually accompanied
        by the urge to vomit. Common causes are early pregnancy, sea and moti
pipeline_tag: sentence-similarity
library_name: sentence-transformers

SentenceTransformer based on cambridgeltl/SapBERT-from-PubMedBERT-fulltext

This is a sentence-transformers model finetuned from cambridgeltl/SapBERT-from-PubMedBERT-fulltext. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: cambridgeltl/SapBERT-from-PubMedBERT-fulltext
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Stevenf232/context_fine-tuned-SapBERT")
# Run inference
sentences = [
    'acromegaly [SEP] This article reports the changes in gallbladder function examined by ultrasonography in 20 Chinese patients with active acromega',
    'Nitroglycerin [SEP] A volatile vasodilator which relieves ANGINA PECTORIS by stimulating GUANYLATE CYCLASE and lowering cytosolic calcium. It is als',
    'Indocyanine Green [SEP] A tricarbocyanine dye that is used diagnostically in liver function tests and to determine blood volume and cardiac output.\n    ',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.2218, 0.3974],
#         [0.2218, 1.0000, 0.5304],
#         [0.3974, 0.5304, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 27,120 training samples
Columns: sentence_0, sentence_1, and label
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1 label
type string string float
details
min: 9 tokens
mean: 27.48 tokens
max: 63 tokens

min: 4 tokens
mean: 24.74 tokens
max: 43 tokens

min: 0.0
mean: 0.21
max: 1.0

	sentence_0	sentence_1	label
type	string	string	float
details	min: 9 tokens mean: 27.48 tokens max: 63 tokens	min: 4 tokens mean: 24.74 tokens max: 43 tokens	min: 0.0 mean: 0.21 max: 1.0

Samples:

sentence_0	sentence_1	label
`toxic to the central nervous system [SEP] Treatment for scabies is usually initiated by general practitioners; most consider lindane (gamma benzene hexachloride) the trea`	`Thyrotoxicosis [SEP] A hypermetabolic syndrome caused by excess THYROID HORMONES which may come from endogenous or exogenous sources. The endogenous`	`0.0`
`cancer [SEP] Doxorubicin is an effective anticancer chemotherapeutic agent known to cause acute and chronic cardiomyopathy. To develop a more`	`Nystagmus, Pathologic [SEP] Involuntary movements of the eye that are divided into two types, jerk and pendular. Jerk nystagmus has a slow phase in one dire`	`0.0`
`doxorubicin [SEP] Doxorubicin is an effective anticancer chemotherapeutic agent known to cause acute and chronic cardiomyopathy. To develop a more`	`Cisplatin [SEP] An inorganic and water-soluble platinum complex. After undergoing hydrolysis, it reacts with DNA to produce both intra and inter`	`0.0`

Loss: ContrastiveLoss with these parameters:

{
    "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
    "margin": 0.5,
    "size_average": true
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 16
per_device_eval_batch_size: 16
fp16: True
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 3
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: None
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
enable_jit_checkpoint: False
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
use_cpu: False
seed: 42
data_seed: None
bf16: False
fp16: True
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: -1
ddp_backend: None
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_for_metrics: []
eval_do_concat_batches: True
auto_find_batch_size: False
full_determinism: False
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
use_cache: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.2950	500	0.0106
0.5900	1000	0.0065
0.8850	1500	0.0058
1.1799	2000	0.0045
1.4749	2500	0.0038
1.7699	3000	0.0036
2.0649	3500	0.0036
2.3599	4000	0.0027
2.6549	4500	0.0027
2.9499	5000	0.0027

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.2.3
Transformers: 5.0.0
PyTorch: 2.9.0+cu128
Accelerate: 1.12.0
Datasets: 4.0.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ContrastiveLoss

@inproceedings{hadsell2006dimensionality,
    author={Hadsell, R. and Chopra, S. and LeCun, Y.},
    booktitle={2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)},
    title={Dimensionality Reduction by Learning an Invariant Mapping},
    year={2006},
    volume={2},
    number={},
    pages={1735-1742},
    doi={10.1109/CVPR.2006.100}
}