--- tags: - sentence-transformers - cross-encoder - reranker - generated_from_trainer - dataset_size:87398 - loss:CrossEntropyLoss base_model: deepvk/USER-bge-m3 pipeline_tag: text-classification library_name: sentence-transformers metrics: - f1_macro - f1_micro - f1_weighted model-index: - name: CrossEncoder based on deepvk/USER-bge-m3 results: - task: type: cross-encoder-softmax-accuracy name: Cross Encoder Softmax Accuracy dataset: name: softmax accuracy eval type: softmax_accuracy_eval metrics: - type: f1_macro value: 0.9715485242270209 name: F1 Macro - type: f1_micro value: 0.9743012183884509 name: F1 Micro - type: f1_weighted value: 0.974262256621189 name: F1 Weighted --- # CrossEncoder based on deepvk/USER-bge-m3 This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [deepvk/USER-bge-m3](https://huggingface.co/deepvk/USER-bge-m3) using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text pair classification. ## Model Details ### Model Description - **Model Type:** Cross Encoder - **Base model:** [deepvk/USER-bge-m3](https://huggingface.co/deepvk/USER-bge-m3) - **Maximum Sequence Length:** 8192 tokens - **Number of Output Labels:** 2 labels ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import CrossEncoder # Download from the πŸ€— Hub model = CrossEncoder("Chimalpopoka/CrossEncoderRanker") # Get scores for pairs of texts pairs = [ ['ПанСль Π°Π»Π»Π΅Ρ€Π³Π΅Π½ΠΎΠ² ΠΏΡ‹Π»ΠΈ β„– 1 IgE (домашняя ΠΏΡ‹Π»ΡŒ (Greer), ΠΊΠ»Π΅Ρ‰-Π΄Π΅Ρ€ΠΌΠ°Ρ‚ΠΎΡ„Π°Π³ ΠΏΠ΅Ρ€ΠΈΠ½Π½Ρ‹ΠΉ, ΠΊΠ»Π΅Ρ‰-Π΄Π΅Ρ€ΠΌΠ°Ρ‚ΠΎΡ„Π°Π³ ΠΌΡƒΡ‡Π½ΠΎΠΉ, Ρ‚Π°Ρ€Π°ΠΊΠ°Π½)', 'БмСсь Π°Π»Π»Π΅Ρ€Π³Π΅Π½ΠΎΠ² ΠΏΡ‹Π»ΠΈ - hm1, Бостав: домашняя ΠΏΡ‹Π»ΡŒ, Dermatophagoides pteronyssinus, Dermatophagoides farinae, Ρ‚Π°Ρ€Π°ΠΊΠ°Π½-прусак, IgE. ΠœΠ΅Ρ‚ΠΎΠ΄: ИЀА'], ['Жидкостная цитология РШМ', 'Жидкостная цитология. ИсслСдованиС соскоба шСйки ΠΌΠ°Ρ‚ΠΊΠΈ ΠΈ Ρ†Π΅Ρ€Π²ΠΈΠΊΠ°Π»ΡŒΠ½ΠΎΠ³ΠΎ ΠΊΠ°Π½Π°Π»Π° (ΠΎΠΊΡ€Π°ΡˆΠΈΠ²Π°Π½ΠΈΠ΅ ΠΏΠΎ ΠŸΠ°ΠΏΠ°Π½ΠΈΠΊΠΎΠ»Π°Ρƒ)'], ['ПосСв Π½Π° Π²ΠΎΠ·Π±ΡƒΠ΄ΠΈΡ‚Π΅Π»Π΅ΠΉ ΠΊΠΈΡˆΠ΅Ρ‡Π½ΠΎΠΉ ΠΈΠ½Ρ„Π΅ΠΊΡ†ΠΈΠΈ (ΡΠ°Π»ΡŒΠΌΠΎΠ½Π΅Π»Π»Ρ‹, ΡˆΠΈΠ³Π΅Π»Π»Ρ‹) с ΠΎΠΏΡ€Π΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ΠΌ Ρ‡ΡƒΠ²ΡΡ‚Π²ΠΈΡ‚Π΅Π»ΡŒΠ½ΠΎΡΡ‚ΠΈ ΠΊ основному спСктру Π°Π½Ρ‚ΠΈΠ±ΠΈΠΎΡ‚ΠΈΠΊΠΎΠ²', 'ПосСв ΠΊΠ°Π»Π° Π½Π° ΠΏΠ°Ρ‚ΠΎΠ³Π΅Π½Π½ΡƒΡŽ Ρ„Π»ΠΎΡ€Ρƒ (дизСнтСрийная ΠΈ тифопаратифозная Π³Ρ€ΡƒΠΏΠΏΡ‹): Π‘ ΠΎΠΏΡ€Π΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ΠΌ Ρ‡ΡƒΠ²ΡΡ‚Π²ΠΈΡ‚Π΅Π»ΡŒΠ½ΠΎΡΡ‚ΠΈ ΠΊ Π°Π½Ρ‚ΠΈΠ±ΠΈΠΎΡ‚ΠΈΠΊΠ°ΠΌ. ΠœΠ΅Ρ‚ΠΎΠ΄: ΠΊΡƒΠ»ΡŒΡ‚ΡƒΡ€Π°Π»ΡŒΠ½Ρ‹ΠΉ'], ['ΠœΠΎΠ»Π΅ΠΊΡƒΠ»ΡΡ€Π½ΠΎ-гСнСтичСскоС исслСдованиС ΠΌΡƒΡ‚Π°Ρ†ΠΈΠΈ Π² Π³Π΅Π½Π΅ V617F (Π·Π°ΠΌΠ΅Π½Π° 617-ΠΎΠΉ аминокислоты с Π²Π°Π»ΠΈΠ½Π° Π½Π° Ρ„Π΅Π½ΠΈΠ»Π°Π»Π°Π½ΠΈΠ½) JAK2 (янус Ρ‚ΠΈΡ€ΠΎΠ·ΠΈΠ½-ΠΊΠΈΠ½Π°Π·Π° Π²Ρ‚ΠΎΡ€ΠΎΠ³ΠΎ Ρ‚ΠΈΠΏΠ° / ΠšΠ°Ρ‡Π΅ΡΡ‚Π²Π΅Π½Π½Π°Ρ ΠΎΡ†Π΅Π½ΠΊΠ° наличия соматичСской ΠΌΡƒΡ‚Π°Ρ†ΠΈΠΈ V617F Π² 14 экзонС Π³Π΅Π½Π° JAK2 (Qualitative assessment of presence of gene JAK2 617F somatic mutation)', 'Анализ ΠΌΡƒΡ‚Π°Ρ†ΠΈΠΈ V617F Π³Π΅Π½Π° JAK2 (Π·Π°ΠΌΠ΅Π½Π° Π²Π°Π»ΠΈΠ½ Π½Π° Ρ„Π΅Π½ΠΈΠ»Π°Π»Π°Π½ΠΈΠ½). ΠœΠ΅Ρ‚ΠΎΠ΄: ПЦР'], ['Π’ΠΎΠ΄ΠΎΡ€ΠΎΠ΄Π½ΠΎ-ΠΌΠ΅Ρ‚Π°Π½ΠΎΠ²Ρ‹ΠΉ Π΄Ρ‹Ρ…Π°Ρ‚Π΅Π»ΡŒΠ½Ρ‹ΠΉ тСст с Π»Π°ΠΊΡ‚ΡƒΠ»ΠΎΠ·ΠΎΠΉ (Π‘Π˜Π‘Π Π’Π•Π‘Π’, синдром ΠΈΠ·Π±Ρ‹Ρ‚ΠΎΡ‡Π½ΠΎΠ³ΠΎ Π±Π°ΠΊΡ‚Π΅Ρ€ΠΈΠ°Π»ΡŒΠ½ΠΎΠ³ΠΎ роста Π² Ρ‚ΠΎΠ½ΠΊΠΎΠΉ кишкС, Π‘Π˜Π‘Π ) (ΡΠ°ΠΌΠΎΡΡ‚ΠΎΡΡ‚Π΅Π»ΡŒΠ½ΠΎΠ΅ взятиС ΠΏΡ€ΠΎΠ±)', 'Π”Ρ‹Ρ…Π°Ρ‚Π΅Π»ΡŒΠ½Ρ‹ΠΉ Π²ΠΎΠ΄ΠΎΡ€ΠΎΠ΄Π½Ρ‹ΠΉ тСст Π½Π° Π‘Π˜Π‘Π '], ] scores = model.predict(pairs) print(scores.shape) # (5, 2) ``` ## Evaluation ### Metrics #### Cross Encoder Softmax Accuracy * Dataset: `softmax_accuracy_eval` * Evaluated with [CESoftmaxAccuracyEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CESoftmaxAccuracyEvaluator) | Metric | Value | |:-------------|:-----------| | **f1_macro** | **0.9715** | | f1_micro | 0.9743 | | f1_weighted | 0.9743 | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 87,398 training samples * Columns: sentence_0, sentence_1, and label * Approximate statistics based on the first 1000 samples: | | sentence_0 | sentence_1 | label | |:--------|:-----------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | sentence_0 | sentence_1 | label | |:---------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | ПанСль Π°Π»Π»Π΅Ρ€Π³Π΅Π½ΠΎΠ² ΠΏΡ‹Π»ΠΈ β„– 1 IgE (домашняя ΠΏΡ‹Π»ΡŒ (Greer), ΠΊΠ»Π΅Ρ‰-Π΄Π΅Ρ€ΠΌΠ°Ρ‚ΠΎΡ„Π°Π³ ΠΏΠ΅Ρ€ΠΈΠ½Π½Ρ‹ΠΉ, ΠΊΠ»Π΅Ρ‰-Π΄Π΅Ρ€ΠΌΠ°Ρ‚ΠΎΡ„Π°Π³ ΠΌΡƒΡ‡Π½ΠΎΠΉ, Ρ‚Π°Ρ€Π°ΠΊΠ°Π½) | БмСсь Π°Π»Π»Π΅Ρ€Π³Π΅Π½ΠΎΠ² ΠΏΡ‹Π»ΠΈ - hm1, Бостав: домашняя ΠΏΡ‹Π»ΡŒ, Dermatophagoides pteronyssinus, Dermatophagoides farinae, Ρ‚Π°Ρ€Π°ΠΊΠ°Π½-прусак, IgE. ΠœΠ΅Ρ‚ΠΎΠ΄: ИЀА | 1 | | Жидкостная цитология РШМ | Жидкостная цитология. ИсслСдованиС соскоба шСйки ΠΌΠ°Ρ‚ΠΊΠΈ ΠΈ Ρ†Π΅Ρ€Π²ΠΈΠΊΠ°Π»ΡŒΠ½ΠΎΠ³ΠΎ ΠΊΠ°Π½Π°Π»Π° (ΠΎΠΊΡ€Π°ΡˆΠΈΠ²Π°Π½ΠΈΠ΅ ΠΏΠΎ ΠŸΠ°ΠΏΠ°Π½ΠΈΠΊΠΎΠ»Π°Ρƒ) | 1 | | ПосСв Π½Π° Π²ΠΎΠ·Π±ΡƒΠ΄ΠΈΡ‚Π΅Π»Π΅ΠΉ ΠΊΠΈΡˆΠ΅Ρ‡Π½ΠΎΠΉ ΠΈΠ½Ρ„Π΅ΠΊΡ†ΠΈΠΈ (ΡΠ°Π»ΡŒΠΌΠΎΠ½Π΅Π»Π»Ρ‹, ΡˆΠΈΠ³Π΅Π»Π»Ρ‹) с ΠΎΠΏΡ€Π΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ΠΌ Ρ‡ΡƒΠ²ΡΡ‚Π²ΠΈΡ‚Π΅Π»ΡŒΠ½ΠΎΡΡ‚ΠΈ ΠΊ основному спСктру Π°Π½Ρ‚ΠΈΠ±ΠΈΠΎΡ‚ΠΈΠΊΠΎΠ² | ПосСв ΠΊΠ°Π»Π° Π½Π° ΠΏΠ°Ρ‚ΠΎΠ³Π΅Π½Π½ΡƒΡŽ Ρ„Π»ΠΎΡ€Ρƒ (дизСнтСрийная ΠΈ тифопаратифозная Π³Ρ€ΡƒΠΏΠΏΡ‹): Π‘ ΠΎΠΏΡ€Π΅Π΄Π΅Π»Π΅Π½ΠΈΠ΅ΠΌ Ρ‡ΡƒΠ²ΡΡ‚Π²ΠΈΡ‚Π΅Π»ΡŒΠ½ΠΎΡΡ‚ΠΈ ΠΊ Π°Π½Ρ‚ΠΈΠ±ΠΈΠΎΡ‚ΠΈΠΊΠ°ΠΌ. ΠœΠ΅Ρ‚ΠΎΠ΄: ΠΊΡƒΠ»ΡŒΡ‚ΡƒΡ€Π°Π»ΡŒΠ½Ρ‹ΠΉ | 1 | * Loss: [CrossEntropyLoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#crossentropyloss) ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `num_train_epochs`: 1 #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 8 - `per_device_eval_batch_size`: 8 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 5e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.0 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `hub_revision`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `liger_kernel_config`: None - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | softmax_accuracy_eval_f1_macro | |:------:|:-----:|:-------------:|:------------------------------:| | 0.0458 | 500 | 0.5378 | - | | 0.0915 | 1000 | 0.2207 | - | | 0.1373 | 1500 | 0.2019 | - | | 0.1831 | 2000 | 0.1981 | 0.9654 | | 0.2288 | 2500 | 0.19 | - | | 0.2746 | 3000 | 0.1703 | - | | 0.3204 | 3500 | 0.217 | - | | 0.3661 | 4000 | 0.1673 | 0.9627 | | 0.4119 | 4500 | 0.1739 | - | | 0.4577 | 5000 | 0.143 | - | | 0.5034 | 5500 | 0.1522 | - | | 0.5492 | 6000 | 0.1545 | 0.9703 | | 0.5950 | 6500 | 0.1353 | - | | 0.6407 | 7000 | 0.1438 | - | | 0.6865 | 7500 | 0.1339 | - | | 0.7323 | 8000 | 0.1355 | 0.9715 | | 0.7780 | 8500 | 0.155 | - | | 0.8238 | 9000 | 0.1256 | - | | 0.8696 | 9500 | 0.1266 | - | | 0.9153 | 10000 | 0.1027 | 0.9715 | ### Framework Versions - Python: 3.12.3 - Sentence Transformers: 5.1.0 - Transformers: 4.53.2 - PyTorch: 2.7.1+cu126 - Accelerate: 1.10.1 - Datasets: 4.0.0 - Tokenizers: 0.21.2 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```