README.md

metadata

tags:
  - sentence-transformers
  - cross-encoder
  - reranker
  - generated_from_trainer
  - dataset_size:43188
  - loss:BinaryCrossEntropyLoss
base_model: cross-encoder/nli-deberta-v3-base
pipeline_tag: text-ranking
library_name: sentence-transformers
metrics:
  - accuracy
  - accuracy_threshold
  - f1
  - f1_threshold
  - precision
  - recall
  - average_precision
model-index:
  - name: CrossEncoder based on cross-encoder/nli-deberta-v3-base
    results:
      - task:
          type: cross-encoder-binary-classification
          name: Cross Encoder Binary Classification
        dataset:
          name: paws val judge
          type: paws-val-judge
        metrics:
          - type: accuracy
            value: 0.9645748987854251
            name: Accuracy
          - type: accuracy_threshold
            value: 0.08707074075937271
            name: Accuracy Threshold
          - type: f1
            value: 0.9604876947392187
            name: F1
          - type: f1_threshold
            value: 0.08707074075937271
            name: F1 Threshold
          - type: precision
            value: 0.9470169189670525
            name: Precision
          - type: recall
            value: 0.9743472285845167
            name: Recall
          - type: average_precision
            value: 0.9870268561433264
            name: Average Precision

CrossEncoder based on cross-encoder/nli-deberta-v3-base

This is a Cross Encoder model finetuned from cross-encoder/nli-deberta-v3-base using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: cross-encoder/nli-deberta-v3-base
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['Route 309 is a Connecticut State Highway in the northwestern Hartford suburbs from Canton to Simsbury .', 'Route 309 runs a Canton State Highway in the northwestern Connecticut suburbs from Hartford to Simsbury .'],
    ['During the competition she lost 50-25 to Zimbabwe , 84-16 to Tanzania , 58-24 to South Africa .', 'During the competition , they lost 50-25 to Zimbabwe , 84-16 to Tanzania , 58-24 to South Africa .'],
    ['The latter study is one of the few prospective demonstrations that environmental stress with high blood pressure and LVH remains associated .', 'The latter study remains one of the few prospective demonstrations that environmental stress with high blood pressure and LVH is associated .'],
    ['The Marignane is located at Marseille Airport in Provence .', 'The Marignane is located in Marseille Provence Airport .'],
    ['Birleffi was of Italian descent and Roman - Catholic in a predominantly Protestant state .', 'Birleffi was of Italian ethnicity and Roman Catholic in a predominantly Protestant state .'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'Route 309 is a Connecticut State Highway in the northwestern Hartford suburbs from Canton to Simsbury .',
    [
        'Route 309 runs a Canton State Highway in the northwestern Connecticut suburbs from Hartford to Simsbury .',
        'During the competition , they lost 50-25 to Zimbabwe , 84-16 to Tanzania , 58-24 to South Africa .',
        'The latter study remains one of the few prospective demonstrations that environmental stress with high blood pressure and LVH is associated .',
        'The Marignane is located in Marseille Provence Airport .',
        'Birleffi was of Italian ethnicity and Roman Catholic in a predominantly Protestant state .',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Binary Classification

• Dataset: paws-val-judge
• Evaluated with CEBinaryClassificationEvaluator

Metric	Value
accuracy	0.9646
accuracy_threshold	0.0871
f1	0.9605
f1_threshold	0.0871
precision	0.947
recall	0.9743
average_precision	0.987

Training Details

Training Dataset

Unnamed Dataset

• Size: 43,188 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 38 characters mean: 114.71 characters max: 200 characters	min: 42 characters mean: 114.33 characters max: 215 characters	min: 0.0 mean: 0.46 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Route 309 is a Connecticut State Highway in the northwestern Hartford suburbs from Canton to Simsbury .	Route 309 runs a Canton State Highway in the northwestern Connecticut suburbs from Hartford to Simsbury .	0.0
  During the competition she lost 50-25 to Zimbabwe , 84-16 to Tanzania , 58-24 to South Africa .	During the competition , they lost 50-25 to Zimbabwe , 84-16 to Tanzania , 58-24 to South Africa .	1.0
  The latter study is one of the few prospective demonstrations that environmental stress with high blood pressure and LVH remains associated .	The latter study remains one of the few prospective demonstrations that environmental stress with high blood pressure and LVH is associated .	1.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	paws-val-judge_average_precision
0.1852	500	0.3758	-
0.3704	1000	0.226	-
0.5556	1500	0.2176	-
0.7407	2000	0.1778	-
0.9259	2500	0.1757	-
1.0	2700	-	0.9826
1.1111	3000	0.1494	-
1.2963	3500	0.1271	-
1.4815	4000	0.1197	-
1.6667	4500	0.1263	-
1.8519	5000	0.116	-
2.0	5400	-	0.9852
2.0370	5500	0.1084	-
2.2222	6000	0.0707	-
2.4074	6500	0.0741	-
2.5926	7000	0.0713	-
2.7778	7500	0.0723	-
2.9630	8000	0.0727	-
3.0	8100	-	0.9870

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.0
• Transformers: 4.57.3
• PyTorch: 2.9.0+cu126
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}