Hugging Face's logo

vimednli
/
vipubmedt5-w_multi-SynPD

Sentence Similarity
sentence-transformers
Safetensors
t5
feature-extraction
Generated from Trainer
dataset_size:92842
loss:MultipleNegativesRankingLoss
Model card Files
xet
Community

SentenceTransformer based on tmnam20/ViPubMedT5

This is a sentence-transformers model finetuned from tmnam20/ViPubMedT5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: tmnam20/ViPubMedT5
  • Maximum Sequence Length: 128 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: T5EncoderModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("vimednli/vipubmedt5-w_multi-SynPD")
# Run inference
sentences = [
    'Việc cấy_ghép các EC và MC có nguồn_gốc từ các tế_bào ES chưa phân_biệt của con_người có khả_năng góp_phần tái_tạo mạch_máu điều_trị và do_đó làm giảm diện_tích nhồi máu sau đột_quỵ .',
    'Việc cấy_ghép các tế_bào mạch_máu có nguồn_gốc từ tế_bào gốc phôi người góp_phần tái_tạo mạch_máu sau đột_quỵ ở chuột .',
    'Những bệnh_nhân ET không bị mất trí_nhớ có nhiều thay_đổi liên_quan đến Alzheimer trong bệnh_lý thần_kinh tau hơn so với nhóm đối_chứng , cho thấy mối liên_hệ giữa bệnh_lý tau và run .',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

  • Size: 92,842 training samples
  • Columns: sentence1 and sentence2
  • Approximate statistics based on the first 1000 samples:
    sentence1 sentence2
    type string string
    details
    • min: 13 tokens
    • mean: 60.83 tokens
    • max: 128 tokens
    • min: 3 tokens
    • mean: 48.05 tokens
    • max: 128 tokens
  • Samples:
    sentence1 sentence2
    Trong nghiên_cứu hiện_tại , hoạt_hoá P21 bằng hoạt_hoá gen do RNA ( RNAa ) gây ra hoạt_động chống khối_u trong ống_nghiệm trên dòng tế_bào u thần_kinh đệm SHG-44 của người . dsRNA nhắm vào vùng khởi_động p21 ( dsP 21 ) đã gây cảm_ứng đáng_kể sự biểu_hiện của p21 ở mức phiên mã và protein , và làm giảm sự biểu_hiện của survivin .
    Kết_quả của nghiên_cứu này cho thấy một sự tương_đồng về trình_tự không mong_đợi của các protein họ GH97 với glycoside hydrolase từ một_số họ khác , có cấu_trúc nếp gấp ( beta / alpha ) 8 của miền xúc_tác và cơ_chế giữ lại quá_trình thuỷ_phân liên_kết glycoside . GH97 là một họ mới của glycoside hydrolase , có liên_quan đến họ siêu alpha-galactosidase.
    MRI là một công_cụ hiệu_quả để dự_đoán đáp_ứng với NAC . Phân giai MRI sau hoá_trị tân_bổ_trợ cho ung_thư vú : sinh_học khối_u có ảnh_hưởng đến độ_chính_xác không ?
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • learning_rate: 3e-05
  • num_train_epochs: 10
  • warmup_ratio: 0.1

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: no
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 3e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 10
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • tp_size: 0
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional

Training Logs

Click to expand
Epoch Step Training Loss
0.0345 100 4.4987
0.0689 200 3.5509
0.1034 300 2.2814
0.1378 400 1.3726
0.1723 500 1.0296
0.2068 600 0.7233
0.2412 700 0.5698
0.2757 800 0.4624
0.3101 900 0.4061
0.3446 1000 0.3497
0.3790 1100 0.2957
0.4135 1200 0.2716
0.4480 1300 0.2456
0.4824 1400 0.2431
0.5169 1500 0.1974
0.5513 1600 0.2078
0.5858 1700 0.2016
0.6203 1800 0.2097
0.6547 1900 0.1855
0.6892 2000 0.1789
0.7236 2100 0.1753
0.7581 2200 0.1629
0.7926 2300 0.1748
0.8270 2400 0.1578
0.8615 2500 0.1452
0.8959 2600 0.1377
0.9304 2700 0.1379
0.9649 2800 0.1439
0.9993 2900 0.1434
1.0338 3000 0.1139
1.0682 3100 0.0966
1.1027 3200 0.1121
1.1371 3300 0.0996
1.1716 3400 0.1095
1.2061 3500 0.1031
1.2405 3600 0.1149
1.2750 3700 0.1239
1.3094 3800 0.0969
1.3439 3900 0.1044
1.3784 4000 0.1141
1.4128 4100 0.0894
1.4473 4200 0.1172
1.4817 4300 0.1009
1.5162 4400 0.0904
1.5507 4500 0.1198
1.5851 4600 0.0956
1.6196 4700 0.1061
1.6540 4800 0.0867
1.6885 4900 0.0908
1.7229 5000 0.1025
1.7574 5100 0.1099
1.7919 5200 0.0932
1.8263 5300 0.0848
1.8608 5400 0.1027
1.8952 5500 0.0851
1.9297 5600 0.0917
1.9642 5700 0.0883
1.9986 5800 0.0931
2.0331 5900 0.0625
2.0675 6000 0.0503
2.1020 6100 0.0627
2.1365 6200 0.0725
2.1709 6300 0.0529
2.2054 6400 0.0591
2.2398 6500 0.0501
2.2743 6600 0.0608
2.3088 6700 0.0616
2.3432 6800 0.0492
2.3777 6900 0.0556
2.4121 7000 0.0744
2.4466 7100 0.0661
2.4810 7200 0.0554
2.5155 7300 0.0615
2.5500 7400 0.0565
2.5844 7500 0.0628
2.6189 7600 0.0527
2.6533 7700 0.069
2.6878 7800 0.0666
2.7223 7900 0.0642
2.7567 8000 0.0601
2.7912 8100 0.0564
2.8256 8200 0.0549
2.8601 8300 0.0552
2.8946 8400 0.0692
2.9290 8500 0.0607
2.9635 8600 0.0537
2.9979 8700 0.0534
3.0324 8800 0.0365
3.0669 8900 0.041
3.1013 9000 0.0405
3.1358 9100 0.0362
3.1702 9200 0.0365
3.2047 9300 0.0451
3.2391 9400 0.0363
3.2736 9500 0.0444
3.3081 9600 0.0349
3.3425 9700 0.0445
3.3770 9800 0.0491
3.4114 9900 0.0429
3.4459 10000 0.0399
3.4804 10100 0.0364
3.5148 10200 0.0429
3.5493 10300 0.0394
3.5837 10400 0.0397
3.6182 10500 0.0406
3.6527 10600 0.038
3.6871 10700 0.0379
3.7216 10800 0.0392
3.7560 10900 0.0395
3.7905 11000 0.0331
3.8249 11100 0.0415
3.8594 11200 0.0421
3.8939 11300 0.0371
3.9283 11400 0.0333
3.9628 11500 0.0352
3.9972 11600 0.0371
4.0317 11700 0.0266
4.0662 11800 0.0288
4.1006 11900 0.0281
4.1351 12000 0.0318
4.1695 12100 0.0256
4.2040 12200 0.0275
4.2385 12300 0.0245
4.2729 12400 0.0295
4.3074 12500 0.0282
4.3418 12600 0.0286
4.3763 12700 0.0231
4.4108 12800 0.03
4.4452 12900 0.0244
4.4797 13000 0.0231
4.5141 13100 0.0222
4.5486 13200 0.027
4.5830 13300 0.0301
4.6175 13400 0.0256
4.6520 13500 0.0325
4.6864 13600 0.0291
4.7209 13700 0.0263
4.7553 13800 0.0215
4.7898 13900 0.0277
4.8243 14000 0.024
4.8587 14100 0.0242
4.8932 14200 0.0259
4.9276 14300 0.0279
4.9621 14400 0.0247
4.9966 14500 0.0285
5.0310 14600 0.0206
5.0655 14700 0.0183
5.0999 14800 0.0161
5.1344 14900 0.019
5.1688 15000 0.0198
5.2033 15100 0.0174
5.2378 15200 0.0157
5.2722 15300 0.0191
5.3067 15400 0.0181
5.3411 15500 0.0165
5.3756 15600 0.018
5.4101 15700 0.0194
5.4445 15800 0.0221
5.4790 15900 0.017
5.5134 16000 0.019
5.5479 16100 0.0166
5.5824 16200 0.0156
5.6168 16300 0.0248
5.6513 16400 0.0189
5.6857 16500 0.0188
5.7202 16600 0.0191
5.7547 16700 0.02
5.7891 16800 0.0157
5.8236 16900 0.0247
5.8580 17000 0.0218
5.8925 17100 0.0191
5.9269 17200 0.0141
5.9614 17300 0.0203
5.9959 17400 0.0169
6.0303 17500 0.0122
6.0648 17600 0.0128
6.0992 17700 0.0151
6.1337 17800 0.0162
6.1682 17900 0.0137
6.2026 18000 0.0124
6.2371 18100 0.0127
6.2715 18200 0.0152
6.3060 18300 0.0151
6.3405 18400 0.0164
6.3749 18500 0.0131
6.4094 18600 0.0155
6.4438 18700 0.0166
6.4783 18800 0.0149
6.5127 18900 0.0165
6.5472 19000 0.0181
6.5817 19100 0.014
6.6161 19200 0.0158
6.6506 19300 0.0171
6.6850 19400 0.0156
6.7195 19500 0.0143
6.7540 19600 0.0142
6.7884 19700 0.0151
6.8229 19800 0.0153
6.8573 19900 0.0141
6.8918 20000 0.0169
6.9263 20100 0.016
6.9607 20200 0.0128
6.9952 20300 0.0145
7.0296 20400 0.0103
7.0641 20500 0.0128
7.0986 20600 0.0088
7.1330 20700 0.0146
7.1675 20800 0.0101
7.2019 20900 0.0145
7.2364 21000 0.0141
7.2708 21100 0.0098
7.3053 21200 0.011
7.3398 21300 0.0117
7.3742 21400 0.0115
7.4087 21500 0.0129
7.4431 21600 0.0121
7.4776 21700 0.0096
7.5121 21800 0.0125
7.5465 21900 0.0115
7.5810 22000 0.0147
7.6154 22100 0.0149
7.6499 22200 0.0133
7.6844 22300 0.0127
7.7188 22400 0.0137
7.7533 22500 0.0113
7.7877 22600 0.0136
7.8222 22700 0.0128
7.8567 22800 0.0127
7.8911 22900 0.0154
7.9256 23000 0.0118
7.9600 23100 0.0118
7.9945 23200 0.0128
8.0289 23300 0.0098
8.0634 23400 0.0103
8.0979 23500 0.0125
8.1323 23600 0.0109
8.1668 23700 0.0083
8.2012 23800 0.0112
8.2357 23900 0.0108
8.2702 24000 0.0113
8.3046 24100 0.0107
8.3391 24200 0.011
8.3735 24300 0.01
8.4080 24400 0.0104
8.4425 24500 0.0099
8.4769 24600 0.0106
8.5114 24700 0.0111
8.5458 24800 0.0111
8.5803 24900 0.0105
8.6147 25000 0.0091
8.6492 25100 0.0128
8.6837 25200 0.0125
8.7181 25300 0.0115
8.7526 25400 0.0119
8.7870 25500 0.0115
8.8215 25600 0.0073
8.8560 25700 0.0107
8.8904 25800 0.012
8.9249 25900 0.0113
8.9593 26000 0.0104
8.9938 26100 0.0124
9.0283 26200 0.0092
9.0627 26300 0.0129
9.0972 26400 0.0094
9.1316 26500 0.0109
9.1661 26600 0.0094
9.2006 26700 0.0098
9.2350 26800 0.0103
9.2695 26900 0.0097
9.3039 27000 0.0106
9.3384 27100 0.0079
9.3728 27200 0.0082
9.4073 27300 0.0095
9.4418 27400 0.0086
9.4762 27500 0.009
9.5107 27600 0.0089
9.5451 27700 0.0102
9.5796 27800 0.0111
9.6141 27900 0.0104
9.6485 28000 0.011
9.6830 28100 0.0096
9.7174 28200 0.0096
9.7519 28300 0.0106
9.7864 28400 0.0076
9.8208 28500 0.0079
9.8553 28600 0.0097
9.8897 28700 0.0083
9.9242 28800 0.0077
9.9586 28900 0.0104
9.9931 29000 0.0107

Framework Versions

  • Python: 3.9.19
  • Sentence Transformers: 4.1.0
  • Transformers: 4.51.3
  • PyTorch: 2.2.0+cu121
  • Accelerate: 0.34.2
  • Datasets: 2.19.1
  • Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss 

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
1
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for vimednli/vipubmedt5-w_multi-SynPD

Base model

tmnam20/ViPubMedT5
Finetuned
(1)
this model

Papers for vimednli/vipubmedt5-w_multi-SynPD