Hugging Face's logo

agentlans
/
deberta-v3-base-zyda-2-v2

Sentence Similarity
sentence-transformers
Safetensors
English
deberta-v2
feature-extraction
Generated from Trainer
dataset_size:1079040
loss:CoSENTLoss
text-embeddings-inference
Model card Files
xet
Community

SentenceTransformer based on agentlans/deberta-v3-base-zyda-2

This is a sentence-transformers model finetuned from agentlans/deberta-v3-base-zyda-2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

This model has been finetuned on the agentlans/sentence-paraphrases dataset.

  • Text were truncated to 500 characters to save VRAM.
  • Two negative examples were generated for each row in the dataset.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: agentlans/deberta-v3-base-zyda-2
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DebertaV2Model 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("agentlans/deberta-v3-base-zyda-2-v2")
# Run inference
sentences = [
    "While holidaying with her partner and his son, Mrs Searle, an administrator, stated: 'If I had been informed, I would have instructed her to desist.'",
    "Mrs Searle, an administrator, who was on holiday with her partner and his son, added: 'If I had known I would have told her to stop.",
    'Mechanical digestion involves breaking down large food pieces into smaller ones that can be acted upon by digestive enzymes.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

  • Size: 1,079,040 training samples
  • Columns: sentence_0, sentence_1, and label
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1 label
    type string string float
    details
    • min: 6 tokens
    • mean: 22.64 tokens
    • max: 75 tokens
    • min: 6 tokens
    • mean: 20.55 tokens
    • max: 71 tokens
    • min: 0.0
    • mean: 0.33
    • max: 1.0
  • Samples:
    sentence_0 sentence_1 label
    Mrs Alper's late spouse, Sam, had an extensive art collection that was also destroyed. The art collection of Mrs Alper's deceased husband, Sam, was also destroyed. 1.0
    Is the 1 x 1 Rubik's cube intended to be comical? What is the least fuve digit decimal number with five significant figures? 0.0
    What is the Prasoon Joshi song that you enjoy the most and what is the rationale behind your preference? What are some interesting facts about human brain? 0.0
  • Loss: CoSENTLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "pairwise_cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: no
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 3
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • eval_use_gather_object: False
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: round_robin

Training Logs

Click to expand
Epoch Step Training Loss
0.0074 500 1.1791
0.0148 1000 0.4056
0.0222 1500 0.2729
0.0297 2000 0.1775
0.0371 2500 0.1513
0.0445 3000 0.1329
0.0519 3500 0.1503
0.0593 4000 0.1325
0.0667 4500 0.1187
0.0741 5000 0.1012
0.0816 5500 0.1504
0.0890 6000 0.1161
0.0964 6500 0.1194
0.1038 7000 0.1172
0.1112 7500 0.1391
0.1186 8000 0.1056
0.1260 8500 0.0697
0.1335 9000 0.1157
0.1409 9500 0.1009
0.1483 10000 0.0996
0.1557 10500 0.1076
0.1631 11000 0.1057
0.1705 11500 0.084
0.1779 12000 0.0711
0.1853 12500 0.1026
0.1928 13000 0.0841
0.2002 13500 0.1149
0.2076 14000 0.0871
0.2150 14500 0.1201
0.2224 15000 0.0851
0.2298 15500 0.073
0.2372 16000 0.0893
0.2447 16500 0.1083
0.2521 17000 0.0824
0.2595 17500 0.0721
0.2669 18000 0.056
0.2743 18500 0.1062
0.2817 19000 0.094
0.2891 19500 0.0887
0.2966 20000 0.0756
0.3040 20500 0.0932
0.3114 21000 0.0718
0.3188 21500 0.067
0.3262 22000 0.0792
0.3336 22500 0.0639
0.3410 23000 0.0987
0.3485 23500 0.0682
0.3559 24000 0.0769
0.3633 24500 0.1255
0.3707 25000 0.0929
0.3781 25500 0.0948
0.3855 26000 0.0983
0.3929 26500 0.1228
0.4004 27000 0.1028
0.4078 27500 0.0856
0.4152 28000 0.1173
0.4226 28500 0.0718
0.4300 29000 0.0964
0.4374 29500 0.0844
0.4448 30000 0.0871
0.4523 30500 0.0943
0.4597 31000 0.1353
0.4671 31500 0.0634
0.4745 32000 0.1263
0.4819 32500 0.1098
0.4893 33000 0.098
0.4967 33500 0.1182
0.5042 34000 0.0818
0.5116 34500 0.1207
0.5190 35000 0.097
0.5264 35500 0.0904
0.5338 36000 0.1011
0.5412 36500 0.1323
0.5486 37000 0.0699
0.5560 37500 0.0803
0.5635 38000 0.0737
0.5709 38500 0.0798
0.5783 39000 0.1348
0.5857 39500 0.0914
0.5931 40000 0.0654
0.6005 40500 0.0729
0.6079 41000 0.0737
0.6154 41500 0.1018
0.6228 42000 0.0809
0.6302 42500 0.0906
0.6376 43000 0.0955
0.6450 43500 0.0759
0.6524 44000 0.1055
0.6598 44500 0.0924
0.6673 45000 0.1027
0.6747 45500 0.0826
0.6821 46000 0.0763
0.6895 46500 0.1035
0.6969 47000 0.0969
0.7043 47500 0.0714
0.7117 48000 0.0826
0.7192 48500 0.0923
0.7266 49000 0.0651
0.7340 49500 0.0901
0.7414 50000 0.1001
0.7488 50500 0.0961
0.7562 51000 0.085
0.7636 51500 0.0633
0.7711 52000 0.0879
0.7785 52500 0.0717
0.7859 53000 0.0589
0.7933 53500 0.0822
0.8007 54000 0.0857
0.8081 54500 0.0994
0.8155 55000 0.0752
0.8230 55500 0.0965
0.8304 56000 0.0776
0.8378 56500 0.089
0.8452 57000 0.0638
0.8526 57500 0.111
0.8600 58000 0.072
0.8674 58500 0.0755
0.8749 59000 0.096
0.8823 59500 0.1205
0.8897 60000 0.0728
0.8971 60500 0.1014
0.9045 61000 0.0987
0.9119 61500 0.0756
0.9193 62000 0.0746
0.9267 62500 0.0992
0.9342 63000 0.0961
0.9416 63500 0.0861
0.9490 64000 0.0723
0.9564 64500 0.0765
0.9638 65000 0.0859
0.9712 65500 0.0839
0.9786 66000 0.085
0.9861 66500 0.1136
0.9935 67000 0.0735
1.0009 67500 0.0791
1.0083 68000 0.0747
1.0157 68500 0.1148
1.0231 69000 0.1022
1.0305 69500 0.0501
1.0380 70000 0.0735
1.0454 70500 0.0734
1.0528 71000 0.0705
1.0602 71500 0.0854
1.0676 72000 0.0858
1.0750 72500 0.0453
1.0824 73000 0.0768
1.0899 73500 0.0949
1.0973 74000 0.1028
1.1047 74500 0.1192
1.1121 75000 0.0754
1.1195 75500 0.0818
1.1269 76000 0.0662
1.1343 76500 0.0659
1.1418 77000 0.0913
1.1492 77500 0.071
1.1566 78000 0.0682
1.1640 78500 0.0858
1.1714 79000 0.0781
1.1788 79500 0.0782
1.1862 80000 0.0722
1.1937 80500 0.0686
1.2011 81000 0.0751
1.2085 81500 0.0611
1.2159 82000 0.1114
1.2233 82500 0.0856
1.2307 83000 0.0789
1.2381 83500 0.0932
1.2456 84000 0.0873
1.2530 84500 0.0691
1.2604 85000 0.0609
1.2678 85500 0.0568
1.2752 86000 0.0797
1.2826 86500 0.0968
1.2900 87000 0.1113
1.2974 87500 0.0936
1.3049 88000 0.091
1.3123 88500 0.0482
1.3197 89000 0.0898
1.3271 89500 0.0766
1.3345 90000 0.0859
1.3419 90500 0.0851
1.3493 91000 0.0695
1.3568 91500 0.0881
1.3642 92000 0.1095
1.3716 92500 0.0676
1.3790 93000 0.094
1.3864 93500 0.0986
1.3938 94000 0.0844
1.4012 94500 0.0929
1.4087 95000 0.0783
1.4161 95500 0.0963
1.4235 96000 0.1003
1.4309 96500 0.0817
1.4383 97000 0.0754
1.4457 97500 0.0858
1.4531 98000 0.0746
1.4606 98500 0.0916
1.4680 99000 0.0738
1.4754 99500 0.0778
1.4828 100000 0.0897
1.4902 100500 0.1028
1.4976 101000 0.0914
1.5050 101500 0.0771
1.5125 102000 0.0716
1.5199 102500 0.1127
1.5273 103000 0.0785
1.5347 103500 0.0868
1.5421 104000 0.118
1.5495 104500 0.0838
1.5569 105000 0.0963
1.5644 105500 0.0579
1.5718 106000 0.0738
1.5792 106500 0.1182
1.5866 107000 0.1025
1.5940 107500 0.0747
1.6014 108000 0.0604
1.6088 108500 0.0607
1.6163 109000 0.0794
1.6237 109500 0.0793
1.6311 110000 0.084
1.6385 110500 0.1315
1.6459 111000 0.0782
1.6533 111500 0.0724
1.6607 112000 0.0864
1.6681 112500 0.0791
1.6756 113000 0.0772
1.6830 113500 0.0923
1.6904 114000 0.0897
1.6978 114500 0.0833
1.7052 115000 0.0819
1.7126 115500 0.0695
1.7200 116000 0.0919
1.7275 116500 0.074
1.7349 117000 0.0893
1.7423 117500 0.1042
1.7497 118000 0.0648
1.7571 118500 0.0965
1.7645 119000 0.0634
1.7719 119500 0.0705
1.7794 120000 0.0928
1.7868 120500 0.0817
1.7942 121000 0.0756
1.8016 121500 0.0769
1.8090 122000 0.0877
1.8164 122500 0.0697
1.8238 123000 0.1095
1.8313 123500 0.1056
1.8387 124000 0.0931
1.8461 124500 0.0772
1.8535 125000 0.0867
1.8609 125500 0.0706
1.8683 126000 0.091
1.8757 126500 0.0751
1.8832 127000 0.0732
1.8906 127500 0.0615
1.8980 128000 0.0947
1.9054 128500 0.1067
1.9128 129000 0.0692
1.9202 129500 0.064
1.9276 130000 0.109
1.9351 130500 0.0843
1.9425 131000 0.0897
1.9499 131500 0.0999
1.9573 132000 0.0866
1.9647 132500 0.083
1.9721 133000 0.0859
1.9795 133500 0.0761
1.9870 134000 0.1089
1.9944 134500 0.1053
2.0018 135000 0.0581
2.0092 135500 0.0781
2.0166 136000 0.1286
2.0240 136500 0.1309
2.0314 137000 0.0476
2.0388 137500 0.0695
2.0463 138000 0.0746
2.0537 138500 0.063
2.0611 139000 0.0816
2.0685 139500 0.0821
2.0759 140000 0.0671
2.0833 140500 0.0865
2.0907 141000 0.0638
2.0982 141500 0.0803
2.1056 142000 0.0872
2.1130 142500 0.0968
2.1204 143000 0.1052
2.1278 143500 0.0554
2.1352 144000 0.1057
2.1426 144500 0.0565
2.1501 145000 0.0798
2.1575 145500 0.098
2.1649 146000 0.0832
2.1723 146500 0.067
2.1797 147000 0.0604
2.1871 147500 0.0808
2.1945 148000 0.0921
2.2020 148500 0.0767
2.2094 149000 0.0856
2.2168 149500 0.0966
2.2242 150000 0.0643
2.2316 150500 0.068
2.2390 151000 0.1007
2.2464 151500 0.0765
2.2539 152000 0.0662
2.2613 152500 0.067
2.2687 153000 0.0547
2.2761 153500 0.0833
2.2835 154000 0.1087
2.2909 154500 0.0868
2.2983 155000 0.0836
2.3058 155500 0.063
2.3132 156000 0.0459
2.3206 156500 0.0771
2.3280 157000 0.0856
2.3354 157500 0.0513
2.3428 158000 0.0584
2.3502 158500 0.0817
2.3577 159000 0.0948
2.3651 159500 0.0945
2.3725 160000 0.0746
2.3799 160500 0.0923
2.3873 161000 0.0933
2.3947 161500 0.08
2.4021 162000 0.1343
2.4095 162500 0.0699
2.4170 163000 0.0861
2.4244 163500 0.0811
2.4318 164000 0.0671
2.4392 164500 0.0877
2.4466 165000 0.0741
2.4540 165500 0.0834
2.4614 166000 0.0966
2.4689 166500 0.0739
2.4763 167000 0.0916
2.4837 167500 0.087
2.4911 168000 0.0974
2.4985 168500 0.0876
2.5059 169000 0.0954
2.5133 169500 0.0936
2.5208 170000 0.0866
2.5282 170500 0.0789
2.5356 171000 0.0932
2.5430 171500 0.094
2.5504 172000 0.0897
2.5578 172500 0.08
2.5652 173000 0.0664
2.5727 173500 0.0807
2.5801 174000 0.1157
2.5875 174500 0.1272
2.5949 175000 0.0843
2.6023 175500 0.067
2.6097 176000 0.084
2.6171 176500 0.0848
2.6246 177000 0.0805
2.6320 177500 0.0828
2.6394 178000 0.1059
2.6468 178500 0.0912
2.6542 179000 0.0683
2.6616 179500 0.0754
2.6690 180000 0.0844
2.6765 180500 0.0824
2.6839 181000 0.0729
2.6913 181500 0.0771
2.6987 182000 0.0993
2.7061 182500 0.0895
2.7135 183000 0.0706
2.7209 183500 0.0731
2.7284 184000 0.0682
2.7358 184500 0.0775
2.7432 185000 0.0956
2.7506 185500 0.0801
2.7580 186000 0.106
2.7654 186500 0.079
2.7728 187000 0.0636
2.7802 187500 0.0819
2.7877 188000 0.0763
2.7951 188500 0.0963
2.8025 189000 0.0714
2.8099 189500 0.0721
2.8173 190000 0.0599
2.8247 190500 0.0998
2.8321 191000 0.0629
2.8396 191500 0.1043
2.8470 192000 0.0973
2.8544 192500 0.1069
2.8618 193000 0.0557
2.8692 193500 0.07
2.8766 194000 0.1252
2.8840 194500 0.0801
2.8915 195000 0.0759
2.8989 195500 0.105
2.9063 196000 0.0806
2.9137 196500 0.0737
2.9211 197000 0.0841
2.9285 197500 0.1031
2.9359 198000 0.0828
2.9434 198500 0.0842
2.9508 199000 0.083
2.9582 199500 0.0745
2.9656 200000 0.0946
2.9730 200500 0.0802
2.9804 201000 0.0847
2.9878 201500 0.0991
2.9953 202000 0.0747

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.1.0
  • Transformers: 4.43.3
  • PyTorch: 2.3.0+cu121
  • Accelerate: 0.33.0
  • Datasets: 3.2.0
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss 

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}
Downloads last month
1
Safetensors
Model size
0.2B params
Tensor type
BF16
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for agentlans/deberta-v3-base-zyda-2-v2

Base model

microsoft/deberta-v3-base
Finetuned
agentlans/deberta-v3-base-zyda-2
Finetuned
(4)
this model
Finetunes
1 model

Dataset used to train agentlans/deberta-v3-base-zyda-2-v2

Paper for agentlans/deberta-v3-base-zyda-2-v2