benjamintli/modernbert-msmarco

SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'OptimizedModule'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("modernbert-msmarco")
# Run inference
queries = [
    "product key windows 8.1 how to find",
]
documents = [
    'If Windows 8.1 came preinstalled on your computer, your Windows 8.1 product key should be on a sticker on your computer or with your documentation. The Windows 8.1 product key is a series of 25 letters and numbers and should look like this: xxxxx-xxxxx-xxxxx-xxxxx-xxxxx.',
    'springer spaniel. n. 1. (Breeds) either of two breeds of large quick-moving spaniels bred to spring game, having a slightly domed head and ears of medium length. The English springer spaniel is the larger and can be of various colours; the Welsh springer spaniel is always a rich red and white. n.',
    'Al Gore not divorced from wife Tipper, confirms relationship with longtime girlfriend. 1  Pucker up! Al Gore planted a wet one on wife Tipper in 2000, during his presidential campaign. Ten years later the couple separated after 40 years of marriage.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.8319, -0.0147, -0.0184]])

Evaluation

Metrics

Information Retrieval

• Dataset: eval
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.795
cosine_accuracy@3	0.9252
cosine_accuracy@5	0.9531
cosine_accuracy@10	0.9735
cosine_precision@1	0.795
cosine_precision@3	0.3084
cosine_precision@5	0.1906
cosine_precision@10	0.0974
cosine_recall@1	0.795
cosine_recall@3	0.9252
cosine_recall@5	0.9531
cosine_recall@10	0.9735
cosine_ndcg@10	0.8909
cosine_mrr@10	0.8636
cosine_map@100	0.8648

Training Details

Training Dataset

msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1

• Dataset: msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 at 84ed2d3
• Size: 477,792 training samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 4 tokens mean: 9.31 tokens max: 24 tokens	min: 17 tokens mean: 81.72 tokens max: 205 tokens

• Samples:

  query	positive
  what is the farthest distance in the universe	Depends on what you mean by seeing. The particle horizon is just the furthest. distance light could have traveled to us since the universe began. That is 93 billion. light years in diameter, or 47 billion light years in any direction, but we can't actually. see anything at that distance.
  what county is laurel ms in	Laurel, MS. Online Offers. Laurel is a city located in Jones County in Mississippi, a state of the United States of America. As of the 2000 census, the city had a total population of 18,393 although a significant population increase has been reported following Hurricane Katrina. Located in southeast Mississippi, southeast of Jackson on Tallahala Creek, Laurel was founded in 1882 as a lumber town. An American Indian reservation is located in nearby Sandersville. Laurel is the principal city of the Laurel Micropolitan Statistical Area.
  how to use a beadloom	How to string your bead loom. To string a loom, attach your nymo thread to one of the small nails at the end of the loom. Run the thread over the metal bars (located on both ends of the loom) and wrap it around one of the small nails on the other end of your loom.ize 8 seed beads are normally to heavy to be used on a loom. The end result would be beadwork that sags in the middle. Every other slow on the metal bar was skipped to accomodate size 8 seed beads. You will not need to do this with seed beads sizes 10-15 that are the correct size beads to use on a bead loom.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 64,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1

• Dataset: msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 at 84ed2d3
• Size: 25,147 evaluation samples
• Columns: query and positive
• Approximate statistics based on the first 1000 samples:

  	query	positive
  type	string	string
  details	min: 4 tokens mean: 9.24 tokens max: 26 tokens	min: 15 tokens mean: 81.64 tokens max: 198 tokens

• Samples:

  query	positive
  how long it take for a pimple to burst	Start doing warm compress over the pimple, so that it gradually gets drained over 2 to 3 days. Do it gently, without giving yourself much pain. Also, apply mupirocin ointment over it twice a day for 3 to 4 days.Read above in detail about dealing with your infected pimple.tart doing warm compress over the pimple, so that it gradually gets drained over 2 to 3 days. Do it gently, without giving yourself much pain. Also, apply mupirocin ointment over it twice a day for 3 to 4 days.
  tularosa population	The Village of Tularosa had a population of 2,677 as of July 1, 2017. Tularosa ranks in the upper quartile for Population Density when compared to the other cities, towns and Census Designated Places (CDPs) in New Mexico. See peer rankings below. The primary coordinate point for Tularosa is located at latitude 33.075 and longitude -106.0173 in Otero County.
  do some people have their blood flowing in reverse direction	As a result, not enough blood flows through the valve. Some valves can have both stenosis and backflow problems. Atresia occurs if a heart valve lacks an opening for blood to pass through. Some people are born with heart valve disease, while others acquire it later in life.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 64,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1024
• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.1
• bf16: True
• eval_strategy: epoch
• per_device_eval_batch_size: 1024
• push_to_hub: True
• hub_model_id: modernbert-msmarco
• load_best_model_at_end: True
• dataloader_num_workers: 4
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 1024
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: epoch
• per_device_eval_batch_size: 1024
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: True
• hub_private_repo: None
• hub_model_id: modernbert-msmarco
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: True
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	eval_cosine_ndcg@10
0.1071	50	4.1149	-	-
0.2141	100	0.5296	-	-
0.3212	150	0.3000	-	-
0.4283	200	0.2463	-	-
0.5353	250	0.2247	-	-
0.6424	300	0.2032	-	-
0.7495	350	0.1923	-	-
0.8565	400	0.1900	-	-
0.9636	450	0.1888	-	-
1.0	467	-	0.1889	0.8768
0.1071	50	0.1866	-	-
0.2141	100	0.1560	-	-
0.3212	150	0.1455	-	-
0.4283	200	0.1377	-	-
0.5353	250	0.1397	-	-
0.6424	300	0.1351	-	-
0.7495	350	0.1355	-	-
0.8565	400	0.1417	-	-
0.9636	450	0.1468	-	-
1.0	467	-	0.1512	0.8909

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.3.0
• Transformers: 5.3.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.7.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}