OloriBern/hotpotqa-bge-m3-2000

CrossEncoder based on BAAI/bge-reranker-v2-m3

This is a Cross Encoder model finetuned from BAAI/bge-reranker-v2-m3 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: BAAI/bge-reranker-v2-m3
• Maximum Sequence Length: 1024 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['Who was the writer of the musical for which Edwin Gagiano received his Best Actor Award for the role of Alf Bueller?', "Back to the 80s (musical). Back to the 80's [sic] is a musical written by Neil Gooding with the original musical adaption made by Scott Copeman.  It was later re-orchestrated and arranged by Brett Foster in 2003, just prior to the Australian Production.  It was originally staged by Neil Gooding Productions Pty Ltd in Sydney, Australia in 2004.  It is a popular show for school productions in the English speaking world."],
    ['Are Stephens Cur and Smooth Collie both herding dogs?', 'Stephens Cur. The Stephens Cur (a.k.a. Stephens\' Stock Cur), is a scent hound that belongs to the Cur dog breed.  They were originally bred by the Stephens family in southeastern Kentucky.  The dogs known as "Little black dog" were bred by generations of that family for over a century.  In 1970, they were recognized as separate and distinct breed of Cur.  The dog is mostly black with white markings, but more than a third white is not permissible.  It is good for hunting raccoon and squirrel, but can also be used to bay wild boar.  They are registered with the United Kennel Club'],
    ['What was the name of the radio telescope the surpassed the Dwingeloo Radio Observatory?', 'Hoopoe-billed ʻakialoa. The hoopoe-billed ʻakialoa, ("Akialoa upupirostris"), was an extinct species of Hawaiian honeycreeper.  Fossil remains have been found of this species in the Hawaiian islands of Kauai and Oahu.  The species specific name, "upupirostris", is derived from the Latin "upupa", hoopoe, and "rostrum", bill, and refers to the long sickle-shaped bill which resembles that of the hoopoe.  The species was apparently slightly larger than others in its genus.  A similar but smaller bird has been discovered but is as yet undescribed from the island of Maui.  The species presumably became extinct after the arrival of humans in Hawaii, and is known only from the fossil record.'],
    ['Are both Robert Stevenson and Am Rong a filmmaker?', 'Will Arnett. William Emerson Arnett ( ; born May 4, 1970) is a Canadian-American actor, voice actor and comedian.  He is best known for his role as George Oscar "Gob" Bluth II in the Fox/Netflix series "Arrested Development" (2003–2006, 2013, 2018); as well as his titular role as BoJack Horseman in the Netflix Original Series of the same name (2014-present).  He has appeared in films such as "Blades of Glory" (2007), "Hot Rod" (2007) and "Teenage Mutant Ninja Turtles" (2014).'],
    ['Who founded this American guitar manufacturer headquartered in Maryland that produced electric baritone guitars?', 'Baritone guitar. The baritone guitar is a guitar with a longer scale length, typically a larger body, and heavier internal bracing, so it can be tuned to a lower pitch.  Gretsch, Fender, Gibson, Ibanez, ESP Guitars, PRS Guitars, Music Man, Danelectro, Schecter, Jerry Jones Guitars, Burns London and many other companies have produced electric baritone guitars since the 1960s, although always in small numbers due to low popularity.  Tacoma, Santa Cruz, Taylor, Martin, Alvarez Guitars and others have made acoustic baritone guitars.'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'Who was the writer of the musical for which Edwin Gagiano received his Best Actor Award for the role of Alf Bueller?',
    [
        "Back to the 80s (musical). Back to the 80's [sic] is a musical written by Neil Gooding with the original musical adaption made by Scott Copeman.  It was later re-orchestrated and arranged by Brett Foster in 2003, just prior to the Australian Production.  It was originally staged by Neil Gooding Productions Pty Ltd in Sydney, Australia in 2004.  It is a popular show for school productions in the English speaking world.",
        'Stephens Cur. The Stephens Cur (a.k.a. Stephens\' Stock Cur), is a scent hound that belongs to the Cur dog breed.  They were originally bred by the Stephens family in southeastern Kentucky.  The dogs known as "Little black dog" were bred by generations of that family for over a century.  In 1970, they were recognized as separate and distinct breed of Cur.  The dog is mostly black with white markings, but more than a third white is not permissible.  It is good for hunting raccoon and squirrel, but can also be used to bay wild boar.  They are registered with the United Kennel Club',
        'Hoopoe-billed ʻakialoa. The hoopoe-billed ʻakialoa, ("Akialoa upupirostris"), was an extinct species of Hawaiian honeycreeper.  Fossil remains have been found of this species in the Hawaiian islands of Kauai and Oahu.  The species specific name, "upupirostris", is derived from the Latin "upupa", hoopoe, and "rostrum", bill, and refers to the long sickle-shaped bill which resembles that of the hoopoe.  The species was apparently slightly larger than others in its genus.  A similar but smaller bird has been discovered but is as yet undescribed from the island of Maui.  The species presumably became extinct after the arrival of humans in Hawaii, and is known only from the fossil record.',
        'Will Arnett. William Emerson Arnett ( ; born May 4, 1970) is a Canadian-American actor, voice actor and comedian.  He is best known for his role as George Oscar "Gob" Bluth II in the Fox/Netflix series "Arrested Development" (2003–2006, 2013, 2018); as well as his titular role as BoJack Horseman in the Netflix Original Series of the same name (2014-present).  He has appeared in films such as "Blades of Glory" (2007), "Hot Rod" (2007) and "Teenage Mutant Ninja Turtles" (2014).',
        'Baritone guitar. The baritone guitar is a guitar with a longer scale length, typically a larger body, and heavier internal bracing, so it can be tuned to a lower pitch.  Gretsch, Fender, Gibson, Ibanez, ESP Guitars, PRS Guitars, Music Man, Danelectro, Schecter, Jerry Jones Guitars, Burns London and many other companies have produced electric baritone guitars since the 1960s, although always in small numbers due to low popularity.  Tacoma, Santa Cruz, Taylor, Martin, Alvarez Guitars and others have made acoustic baritone guitars.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Binary Classification

• Datasets: validation and train_subset
• Evaluated with CEBinaryClassificationEvaluator

Metric	validation	train_subset
accuracy	0.99	0.942
accuracy_threshold	0.3649	0.7961
f1	0.99	0.9437
f1_threshold	0.3649	0.6639
precision	0.9933	0.9346
recall	0.9867	0.9529
average_precision	0.9991	0.9847

Training Details

Training Dataset

Unnamed Dataset

• Size: 8,000 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 23 characters mean: 104.78 characters max: 498 characters	min: 79 characters mean: 541.67 characters max: 2408 characters	min: 0.0 mean: 0.5 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Who was the writer of the musical for which Edwin Gagiano received his Best Actor Award for the role of Alf Bueller?	Back to the 80s (musical). Back to the 80's [sic] is a musical written by Neil Gooding with the original musical adaption made by Scott Copeman. It was later re-orchestrated and arranged by Brett Foster in 2003, just prior to the Australian Production. It was originally staged by Neil Gooding Productions Pty Ltd in Sydney, Australia in 2004. It is a popular show for school productions in the English speaking world.	1.0
  Are Stephens Cur and Smooth Collie both herding dogs?	Stephens Cur. The Stephens Cur (a.k.a. Stephens' Stock Cur), is a scent hound that belongs to the Cur dog breed. They were originally bred by the Stephens family in southeastern Kentucky. The dogs known as "Little black dog" were bred by generations of that family for over a century. In 1970, they were recognized as separate and distinct breed of Cur. The dog is mostly black with white markings, but more than a third white is not permissible. It is good for hunting raccoon and squirrel, but can also be used to bay wild boar. They are registered with the United Kennel Club	1.0
  What was the name of the radio telescope the surpassed the Dwingeloo Radio Observatory?	Hoopoe-billed ʻakialoa. The hoopoe-billed ʻakialoa, ("Akialoa upupirostris"), was an extinct species of Hawaiian honeycreeper. Fossil remains have been found of this species in the Hawaiian islands of Kauai and Oahu. The species specific name, "upupirostris", is derived from the Latin "upupa", hoopoe, and "rostrum", bill, and refers to the long sickle-shaped bill which resembles that of the hoopoe. The species was apparently slightly larger than others in its genus. A similar but smaller bird has been discovered but is as yet undescribed from the island of Maui. The species presumably became extinct after the arrival of humans in Hawaii, and is known only from the fossil record.	0.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 2
• per_device_eval_batch_size: 2
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	validation_average_precision	train_subset_average_precision
0.025	100	0.9994	0.9911
0.05	200	0.9983	0.9895
0.075	300	0.9981	0.9887
0.1	400	0.9991	0.9847

Framework Versions

• Python: 3.11.13
• Sentence Transformers: 5.2.0
• Transformers: 4.44.2
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}