rezarahim/bge-finetuned-reranker

SentenceTransformer based on BAAI/bge-reranker-base

This is a sentence-transformers model finetuned from BAAI/bge-reranker-base on the train dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-reranker-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • train

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("rezarahim/bge-finetuned-reranker")
# Run inference
sentences = [
    "What are the potential risks associated with the company's acquisitions and strategic investments?",
    " The potential risks include impairment of the company's ability to grow its business, develop new products, or sell its products, as well as the possibility of regulatory conditions reducing the value of the acquisition, volatility in results, losses up to the value of the investment, and impairment losses due to the failure of the invested companies.",
    ' $7,280.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: bge-base-en
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0056
cosine_accuracy@3	0.0506
cosine_accuracy@5	0.0899
cosine_accuracy@10	0.1685
cosine_precision@1	0.0056
cosine_precision@3	0.0169
cosine_precision@5	0.018
cosine_precision@10	0.0169
cosine_recall@1	0.0056
cosine_recall@3	0.0506
cosine_recall@5	0.0899
cosine_recall@10	0.1685
cosine_ndcg@10	0.0728
cosine_mrr@10	0.0442
cosine_map@100	0.0749
dot_accuracy@1	0.0169
dot_accuracy@3	0.0337
dot_accuracy@5	0.0843
dot_accuracy@10	0.1854
dot_precision@1	0.0169
dot_precision@3	0.0112
dot_precision@5	0.0169
dot_precision@10	0.0185
dot_recall@1	0.0169
dot_recall@3	0.0337
dot_recall@5	0.0843
dot_recall@10	0.1854
dot_ndcg@10	0.0789
dot_mrr@10	0.0478
dot_map@100	0.0757

Training Details

Training Dataset

train

• Dataset: train
• Size: 178 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 178 samples:

  	anchor	positive
  type	string	string
  details	min: 12 tokens mean: 23.55 tokens max: 50 tokens	min: 3 tokens mean: 42.22 tokens max: 135 tokens

• Samples:

  anchor	positive
  What is the publication date of the NVIDIA Corporation Annual Report 2024?	February 21st, 2024
  What is the filing date of the 10-K report for NVIDIA Corporation in 2004?	The filing dates of the 10-K reports for NVIDIA Corporation in 2004 are May 20th, March 29th, and April 25th.
  What is the purpose of the section of the filing that requires the registrant to indicate whether it has submitted electronically every Interactive Data File required to be submitted during the preceding 12 months?	The purpose of this section is to comply with Rule 405 of Regulation S-T, which requires the registrant to submit electronic files for certain financial information.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 16
• learning_rate: 2e-05
• num_train_epochs: 25
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• load_best_model_at_end: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 25
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	bge-base-en_dot_map@100
0	0	-	0.0362
0.7111	2	-	0.0369
1.7778	5	-	0.0539
2.8444	8	-	0.0393
3.5556	10	2.0824	-
3.9111	11	-	0.0559
4.9778	14	-	0.0632
5.6889	16	-	0.08
6.7556	19	-	0.0692
7.1111	20	1.2812	-
7.8222	22	-	0.0627
8.8889	25	-	0.0623
9.9556	28	-	0.0692
10.6667	30	1.0855	0.0884
11.7333	33	-	0.0754
12.8	36	-	0.0607
13.8667	39	-	0.0725
14.2222	40	0.8978	-
14.9333	42	-	0.0747
16.0	45	-	0.0766
16.7111	47	-	0.0756
17.7778	50	0.8563	0.0757

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.5.1+cu121
• Accelerate: 1.2.1
• Datasets: 3.1.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}