Hugging Face's logo

galkinv42
/
sergeyzh_BERTA-final-finetuned-duplicates

Sentence Similarity
sentence-transformers
Safetensors
bert
feature-extraction
dense
Generated from Trainer
dataset_size:6921
loss:CosineSimilarityLoss
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community

SentenceTransformer based on sergeyzh/BERTA

This is a sentence-transformers model finetuned from sergeyzh/BERTA on the duplicates-checker-finetuning-preview dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: sergeyzh/BERTA
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity
  • Training Dataset:
    • duplicates-checker-finetuning-preview

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "USSD-\u043a\u043e\u043c\u0430\u043d\u0434\u0430 \u0434\u043b\u044f \u043f\u0440\u043e\u0432\u0435\u0440\u043a\u0438 \u0431\u0430\u043b\u0430\u043d\u0441\u0430 \u0421\u0431\u0435\u0440\u041c\u043e\u0431\u0430\u0439\u043b - *100#.",
]
documents = [
    'Чтобы узнать баланс СберМобайл, наберите *100#.',
    'statement_statement',
    'СберМобайл',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.9729, 0.2598, 0.0023]])

Evaluation

Metrics

Binary Classification

Metric binary-sts-validation binary-sts-test
cosine_accuracy 0.9156 0.9065
cosine_accuracy_threshold 0.6416 0.684
cosine_f1 0.9197 0.9095
cosine_f1_threshold 0.6109 0.6464
cosine_precision 0.8819 0.881
cosine_recall 0.9609 0.94
cosine_ap 0.9155 0.9208
cosine_mcc 0.8345 0.8148

Training Details

Training Dataset

duplicates-checker-finetuning-preview

  • Dataset: duplicates-checker-finetuning-preview
  • Size: 6,921 training samples
  • Columns: sentence1, sentence2, label, task_type, product, and stratify_col
  • Approximate statistics based on the first 1000 samples:
    sentence1 sentence2 label task_type product stratify_col
    type string string int string string string
    details
    • min: 5 tokens
    • mean: 19.33 tokens
    • max: 45 tokens
    • min: 6 tokens
    • mean: 20.3 tokens
    • max: 44 tokens
    • 0: ~49.60%
    • 1: ~50.40%
    • min: 5 tokens
    • mean: 5.0 tokens
    • max: 5 tokens
    • min: 5 tokens
    • mean: 10.06 tokens
    • max: 17 tokens
    • min: 11 tokens
    • mean: 16.06 tokens
    • max: 23 tokens
  • Samples:
    sentence1 sentence2 label task_type product stratify_col
    Облигации Федерального Займа выпускает Министерство финансов РФ, а не Центральный Банк. Облигации Федерального Займа выпускает Министерство финансов РФ, а не СберБанк. 0 correction_correction Облигации 0_correction_correction_Облигации
    Льгота на долгосрочное владение паями ОПИФ действует при владении более 3 лет, а не 1 года. Лимит дохода для ЛДВ по ОПИФ составляет 3 млн рублей за каждый год владения, а не 1 млн. 0 correction_correction Открытый паевой инвестиционный фонд 0_correction_correction_Открытый паевой инвестиционный фонд
    Продажа паев ЗПИФ на бирже не требует поиска покупателя, в отличие от продажи по договору купли-продажи. Потенциальный доход от фонда Современный 8 включает рентный доход и доход от роста стоимости, а не только рентный. 0 correction_correction Закрытый паевой инвестиционный фонд 0_correction_correction_Закрытый паевой инвестиционный фонд
  • Loss: CosineSimilarityLoss with these parameters:
    {
    "loss_fct": "torch.nn.modules.loss.MSELoss"
}

Evaluation Dataset

duplicates-checker-finetuning-preview

  • Dataset: duplicates-checker-finetuning-preview
  • Size: 865 evaluation samples
  • Columns: sentence1, sentence2, label, task_type, product, and stratify_col
  • Approximate statistics based on the first 865 samples:
    sentence1 sentence2 label task_type product stratify_col
    type string string int string string string
    details
    • min: 7 tokens
    • mean: 19.34 tokens
    • max: 37 tokens
    • min: 8 tokens
    • mean: 20.46 tokens
    • max: 39 tokens
    • 0: ~49.71%
    • 1: ~50.29%
    • min: 5 tokens
    • mean: 5.0 tokens
    • max: 5 tokens
    • min: 5 tokens
    • mean: 9.94 tokens
    • max: 17 tokens
    • min: 11 tokens
    • mean: 15.94 tokens
    • max: 23 tokens
  • Samples:
    sentence1 sentence2 label task_type product stratify_col
    Какой тариф Сбера подходит для начинающих инвесторов на ИИС-3? Какой тарифный план Сбера рекомендован для новичков, использующих ИИС-3? 1 question_question Индивидуальный инвестиционный счёт 1_question_question_Индивидуальный инвестиционный счёт
    Какие типы кредитных карт Сбера вы предлагаете, и какие преимущества у каждой из них? Расскажите о видах Кредитных СберКарт и их плюсах. 1 question_question Кредитная СберКарта 1_question_question_Кредитная СберКарта
    При отсутствии трудовой книжки стаж подтверждается справками из архива. При отсутствии трудовой книжки стаж подтверждается устными показаниями свидетелей. 0 statement_statement Перевод пенсии 0_statement_statement_Перевод пенсии
  • Loss: CosineSimilarityLoss with these parameters:
    {
    "loss_fct": "torch.nn.modules.loss.MSELoss"
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • learning_rate: 1.2006506775681832e-05
  • weight_decay: 0.04243902303817388
  • num_train_epochs: 50
  • warmup_ratio: 0.27192485622024914
  • load_best_model_at_end: True

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 1.2006506775681832e-05
  • weight_decay: 0.04243902303817388
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 50
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.27192485622024914
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • hub_revision: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • liger_kernel_config: None
  • eval_use_gather_object: False
  • average_tokens_across_devices: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Click to expand
Epoch Step Training Loss Validation Loss binary-sts-validation_cosine_ap binary-sts-test_cosine_ap
0.2304 50 0.2427 - - -
0.4608 100 0.2435 0.2850 0.8052 -
0.6912 150 0.2318 - - -
0.9217 200 0.2341 0.2740 0.8085 -
1.1521 250 0.2303 - - -
1.3825 300 0.2277 0.2554 0.8147 -
1.6129 350 0.2239 - - -
1.8433 400 0.2048 0.2293 0.8187 -
2.0737 450 0.1955 - - -
2.3041 500 0.1913 0.2015 0.8221 -
2.5346 550 0.1878 - - -
2.7650 600 0.1743 0.1771 0.8230 -
2.9954 650 0.1714 - - -
3.2258 700 0.1679 0.1575 0.8183 -
3.4562 750 0.153 - - -
3.6866 800 0.1437 0.1449 0.8182 -
3.9171 850 0.1421 - - -
4.1475 900 0.1318 0.1371 0.8195 -
4.3779 950 0.1345 - - -
4.6083 1000 0.1265 0.1307 0.8263 -
4.8387 1050 0.1282 - - -
5.0691 1100 0.1245 0.1258 0.8399 -
5.2995 1150 0.1132 - - -
5.5300 1200 0.1142 0.1208 0.8432 -
5.7604 1250 0.1138 - - -
5.9908 1300 0.1119 0.1162 0.8512 -
6.2212 1350 0.1034 - - -
6.4516 1400 0.1034 0.1122 0.8584 -
6.6820 1450 0.1026 - - -
6.9124 1500 0.0985 0.1083 0.8605 -
7.1429 1550 0.0905 - - -
7.3733 1600 0.0912 0.1049 0.8690 -
7.6037 1650 0.0869 - - -
7.8341 1700 0.0876 0.1018 0.8716 -
8.0645 1750 0.0884 - - -
8.2949 1800 0.0833 0.0980 0.8801 -
8.5253 1850 0.0734 - - -
8.7558 1900 0.0764 0.0956 0.8809 -
8.9862 1950 0.0786 - - -
9.2166 2000 0.0728 0.0928 0.8831 -
9.4470 2050 0.0683 - - -
9.6774 2100 0.0674 0.0907 0.8880 -
9.9078 2150 0.0675 - - -
10.1382 2200 0.0604 0.0886 0.8902 -
10.3687 2250 0.0611 - - -
10.5991 2300 0.0584 0.0864 0.8956 -
10.8295 2350 0.0588 - - -
11.0599 2400 0.0624 0.0847 0.9026 -
11.2903 2450 0.0505 - - -
11.5207 2500 0.0513 0.0845 0.8974 -
11.7512 2550 0.0556 - - -
11.9816 2600 0.053 0.0813 0.9021 -
12.2120 2650 0.0445 - - -
12.4424 2700 0.0471 0.0812 0.9044 -
12.6728 2750 0.0446 - - -
12.9032 2800 0.046 0.0804 0.9006 -
13.1336 2850 0.0435 - - -
13.3641 2900 0.0367 0.0800 0.9054 -
13.5945 2950 0.0396 - - -
13.8249 3000 0.0425 0.0796 0.9036 -
14.0553 3050 0.0398 - - -
14.2857 3100 0.0299 0.0772 0.9109 -
14.5161 3150 0.0357 - - -
14.7465 3200 0.0376 0.0762 0.9085 -
14.9770 3250 0.0334 - - -
15.2074 3300 0.0307 0.0765 0.9099 -
15.4378 3350 0.0283 - - -
15.6682 3400 0.0314 0.0765 0.9134 -
15.8986 3450 0.0335 - - -
16.1290 3500 0.0265 0.0749 0.9114 -
16.3594 3550 0.0233 - - -
16.5899 3600 0.0254 0.0754 0.9174 -
16.8203 3650 0.0288 - - -
17.0507 3700 0.0261 0.0743 0.9196 -
17.2811 3750 0.0238 - - -
17.5115 3800 0.0222 0.0748 0.9171 -
17.7419 3850 0.025 - - -
17.9724 3900 0.0252 0.0743 0.9181 -
18.2028 3950 0.0197 - - -
18.4332 4000 0.019 0.0736 0.9195 -
18.6636 4050 0.021 - - -
18.8940 4100 0.0222 0.0731 0.9229 -
19.1244 4150 0.0202 - - -
19.3548 4200 0.0211 0.0740 0.9191 -
19.5853 4250 0.0169 - - -
19.8157 4300 0.0174 0.0745 0.9200 -
20.0461 4350 0.0177 - - -
20.2765 4400 0.0168 0.0736 0.9155 -
20.5069 4450 0.0182 - - -
20.7373 4500 0.0151 0.0740 0.9154 -
20.9677 4550 0.0163 - - -
21.1982 4600 0.0146 0.0740 0.9180 -
21.4286 4650 0.0128 - - -
21.6590 4700 0.0154 0.0734 0.9196 -
21.8894 4750 0.0149 - - -
22.1198 4800 0.0147 0.0743 0.9175 -
22.3502 4850 0.0132 - - -
22.5806 4900 0.0142 0.0745 0.9189 -
22.8111 4950 0.0152 - - -
23.0415 5000 0.013 0.0734 0.9261 -
23.2719 5050 0.0118 - - -
23.5023 5100 0.0119 0.0739 0.9265 -
23.7327 5150 0.012 - - -
23.9631 5200 0.0123 0.0738 0.9246 -
24.1935 5250 0.0131 - - -
24.4240 5300 0.0115 0.0725 0.9264 -
24.6544 5350 0.0116 - - -
24.8848 5400 0.011 0.0731 0.9258 -
25.1152 5450 0.0108 - - -
25.3456 5500 0.0112 0.0728 0.9276 -
25.5760 5550 0.0119 - - -
25.8065 5600 0.0084 0.0732 0.9267 -
26.0369 5650 0.0108 - - -
26.2673 5700 0.0105 0.0734 0.9296 -
26.4977 5750 0.0083 - - -
26.7281 5800 0.0102 0.0733 0.9305 -
26.9585 5850 0.0102 - - -
27.1889 5900 0.0074 0.0731 0.9279 -
27.4194 5950 0.0086 - - -
27.6498 6000 0.0091 0.0741 0.9253 -
27.8802 6050 0.0105 - - -
28.1106 6100 0.0098 0.0738 0.9277 -
28.3410 6150 0.0079 - - -
28.5714 6200 0.009 0.0723 0.9319 -
28.8018 6250 0.0082 - - -
29.0323 6300 0.009 0.0727 0.9302 -
29.2627 6350 0.0092 - - -
29.4931 6400 0.0078 0.0731 0.9348 -
29.7235 6450 0.0079 - - -
29.9539 6500 0.0091 0.0734 0.9361 -
30.1843 6550 0.0091 - - -
30.4147 6600 0.0069 0.0735 0.9380 -
30.6452 6650 0.0075 - - -
30.8756 6700 0.0075 0.0731 0.9384 -
31.1060 6750 0.007 - - -
31.3364 6800 0.0064 0.0739 0.9365 -
31.5668 6850 0.0083 - - -
31.7972 6900 0.0076 0.0732 0.9373 -
32.0276 6950 0.0073 - - -
32.2581 7000 0.0075 0.0740 0.9403 -
32.4885 7050 0.0068 - - -
32.7189 7100 0.0083 0.0730 0.9399 -
32.9493 7150 0.0053 - - -
33.1797 7200 0.0076 0.0725 0.9387 -
33.4101 7250 0.0055 - - -
33.6406 7300 0.007 0.0728 0.9396 -
33.8710 7350 0.0071 - - -
34.1014 7400 0.0058 0.0736 0.9396 -
34.3318 7450 0.0063 - - -
34.5622 7500 0.0066 0.0735 0.9396 -
34.7926 7550 0.0068 - - -
35.0230 7600 0.0056 0.0738 0.9388 -
35.2535 7650 0.0074 - - -
35.4839 7700 0.0061 0.0738 0.9392 -
35.7143 7750 0.0062 - - -
35.9447 7800 0.0054 0.0733 0.9396 -
36.1751 7850 0.0058 - - -
36.4055 7900 0.0061 0.0733 0.9397 -
36.6359 7950 0.0052 - - -
36.8664 8000 0.0062 0.0731 0.9396 -
37.0968 8050 0.0051 - - -
37.3272 8100 0.0066 0.0733 0.9395 -
37.5576 8150 0.0049 - - -
37.7880 8200 0.0051 0.0727 0.9391 -
38.0184 8250 0.0046 - - -
38.2488 8300 0.0056 0.0732 0.9383 -
38.4793 8350 0.0039 - - -
38.7097 8400 0.0047 0.0734 0.9389 -
38.9401 8450 0.0057 - - -
39.1705 8500 0.0064 0.0740 0.9402 -
39.4009 8550 0.0049 - - -
39.6313 8600 0.0057 0.0742 0.9409 -
39.8618 8650 0.0049 - - -
40.0922 8700 0.0057 0.0740 0.9414 -
40.3226 8750 0.0056 - - -
40.5530 8800 0.0043 0.0742 0.9408 -
40.7834 8850 0.0046 - - -
41.0138 8900 0.0051 0.0740 0.9409 -
41.2442 8950 0.0043 - - -
41.4747 9000 0.0046 0.0742 0.9410 -
41.7051 9050 0.0059 - - -
41.9355 9100 0.0044 0.0739 0.9409 -
42.1659 9150 0.0049 - - -
42.3963 9200 0.0048 0.0738 0.9418 -
42.6267 9250 0.0047 - - -
42.8571 9300 0.0036 0.0744 0.9416 -
43.0876 9350 0.0041 - - -
43.3180 9400 0.0049 0.0745 0.9416 -
43.5484 9450 0.0047 - - -
43.7788 9500 0.0048 0.0742 0.9415 -
44.0092 9550 0.0038 - - -
44.2396 9600 0.005 0.0741 0.9416 -
44.4700 9650 0.0045 - - -
44.7005 9700 0.004 0.0743 0.9416 -
44.9309 9750 0.0038 - - -
45.1613 9800 0.0042 0.0739 0.9416 -
45.3917 9850 0.005 - - -
45.6221 9900 0.0051 0.0742 0.9418 -
45.8525 9950 0.004 - - -
46.0829 10000 0.004 0.0744 0.9418 -
46.3134 10050 0.0035 - - -
46.5438 10100 0.0041 0.0743 0.9420 -
46.7742 10150 0.0041 - - -
47.0046 10200 0.0063 0.0744 0.9421 -
47.2350 10250 0.0039 - - -
47.4654 10300 0.0044 0.0744 0.9421 -
47.6959 10350 0.0043 - - -
47.9263 10400 0.0034 0.0744 0.9423 -
48.1567 10450 0.0048 - - -
48.3871 10500 0.0033 0.0744 0.9424 -
48.6175 10550 0.0048 - - -
48.8479 10600 0.0041 0.0745 0.9423 -
49.0783 10650 0.0035 - - -
49.3088 10700 0.0036 0.0744 0.9423 -
49.5392 10750 0.0037 - - -
49.7696 10800 0.0051 0.0744 0.9423 -
50.0 10850 0.0036 - - -
-1 -1 - - 0.9155 0.9208
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.12.7
  • Sentence Transformers: 5.0.0
  • Transformers: 4.54.1
  • PyTorch: 2.5.1
  • Accelerate: 1.9.0
  • Datasets: 4.0.0
  • Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
Downloads last month
1
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
Sentence Similarity
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for galkinv42/sergeyzh_BERTA-final-finetuned-duplicates

Base model

cointegrated/LaBSE-en-ru
Finetuned
sergeyzh/LaBSE-ru-turbo
Finetuned
sergeyzh/BERTA
Finetuned
(4)
this model

Paper for galkinv42/sergeyzh_BERTA-final-finetuned-duplicates

Evaluation results