PrasannSinghal/ModernBERT-base-DPR-8e-05

SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: answerdotai/ModernBERT-base
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("PrasannSinghal/ModernBERT-base-DPR-8e-05")
# Run inference
queries = [
    "do bond funds pay dividends",
]
documents = [
    "A bond fund or debt fund is a fund that invests in bonds, or other debt securities. Bond funds can be contrasted with stock funds and money funds. Bond funds typically pay periodic dividends that include interest payments on the fund's underlying securities plus periodic realized capital appreciation. Bond funds typically pay higher dividends than CDs and money market accounts. Most bond funds pay out dividends more frequently than individual bonds.",
    'You would have $71,200 paying out $1,687 in annual dividends. That is about $4.62 for working up in the morning. Interestingly enough, that 2.37% yield is at a low point because The Wellington Fund is a â\x80\x9cbalanced fundâ\x80\x9d meaning that it holds a combination of stocks and bonds.',
    "If a cavity is causing the toothache, your dentist will fill the cavity or possibly extract the tooth, if necessary. A root canal might be needed if the cause of the toothache is determined to be an infection of the tooth's nerve. Bacteria that have worked their way into the inner aspects of the tooth cause such an infection. An antibiotic may be prescribed if there is fever or swelling of the jaw.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.7494,  0.5887, -0.0276]])

Evaluation

Metrics

Triplet

• Dataset: msmarco-co-condenser-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.971

Training Details

Training Dataset

msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1

• Dataset: msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 at 84ed2d3
• Size: 1,250,000 training samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative
  type	string	string	string
  details	min: 4 tokens mean: 9.26 tokens max: 34 tokens	min: 17 tokens mean: 79.14 tokens max: 222 tokens	min: 24 tokens mean: 80.09 tokens max: 436 tokens

• Samples:

  query	positive	negative
  what is the meaning of menu planning	Menu planning is the selection of a menu for an event. Such as picking out the dinner for your wedding or even a meal at a Birthday Party. Menu planning is when you are preparing a calendar of meals and you have to sit down and decide what meat and veggies you want to serve on each certain day.	Menu Costs. In economics, a menu cost is the cost to a firm resulting from changing its prices. The name stems from the cost of restaurants literally printing new menus, but economists use it to refer to the costs of changing nominal prices in general.
  how old is brett butler	Brett Butler is 59 years old. To be more precise (and nerdy), the current age as of right now is 21564 days or (even more geeky) 517536 hours. That's a lot of hours!	Passed in: St. John's, Newfoundland and Labrador, Canada. Passed on: 16/07/2016. Published in the St. John's Telegram. Passed away suddenly at the Health Sciences Centre surrounded by his loving family, on July 16, 2016 Robert (Bobby) Joseph Butler, age 52 years. Predeceased by his special aunt Geri Murrin and uncle Mike Mchugh; grandparents Joe and Margaret Murrin and Jack and Theresa Butler.
  when was the last navajo treaty sign?	In Executive Session, Senate of the United States, July 25, 1868. Resolved, (two-thirds of the senators present concurring,) That the Senate advise and consent to the ratification of the treaty between the United States and the Navajo Indians, concluded at Fort Sumner, New Mexico, on the first day of June, 1868.	Share Treaty of Greenville. The Treaty of Greenville was signed August 3, 1795, between the United States, represented by Gen. Anthony Wayne, and chiefs of the Indian tribes located in the Northwest Territory, including the Wyandots, Delawares, Shawnees, Ottawas, Miamis, and others.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "gather_across_devices": false
  }
  

Evaluation Dataset

msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1

• Dataset: msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 at 84ed2d3
• Size: 1,000 evaluation samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative
  type	string	string	string
  details	min: 4 tokens mean: 9.2 tokens max: 27 tokens	min: 21 tokens mean: 80.44 tokens max: 241 tokens	min: 23 tokens mean: 80.38 tokens max: 239 tokens

• Samples:

  query	positive	negative
  what county is holly springs nc in	Holly Springs, North Carolina. Holly Springs is a town in Wake County, North Carolina, United States. As of the 2010 census, the town population was 24,661, over 2½ times its population in 2000. Contents.	The Mt. Holly Springs Park & Resort. One of the numerous trolley routes that carried people around the county at the turn of the century was the Carlisle & Mt. Holly Railway Company. The “Holly Trolley” as it came to be known was put into service by Patricio Russo and made its first run on May 14, 1901.
  how long does nyquil stay in your system	In order to understand exactly how long Nyquil lasts, it is absolutely vital to learn about the various ingredients in the drug. One of the ingredients found in Nyquil is Doxylamine, which is an antihistamine. This specific medication has a biological half-life or 6 to 12 hours. With this in mind, it is possible for the drug to remain in the system for a period of 12 to 24 hours. It should be known that the specifics will depend on a wide variety of different factors, including your age and metabolism.	I confirmed that NyQuil is about 10% alcohol, a higher content than most domestic beers. When I asked about the relatively high proof, I was told that the alcohol dilutes the active ingredients. The alcohol free version is there for customers with addiction issues.. also found that in that version there is twice the amount of DXM. When I asked if I could speak to a chemist or scientist, I was told they didn't have anyone who fit that description there. It’s been eight years since I kicked NyQuil. I've been sober from alcohol for four years.
  what are mineral water	1 Mineral water – water from a mineral spring that contains various minerals, such as salts and sulfur compounds. 2 It comes from a source tapped at one or more bore holes or spring, and originates from a geologically and physically protected underground water source. Mineral water – water from a mineral spring that contains various minerals, such as salts and sulfur compounds. 2 It comes from a source tapped at one or more bore holes or spring, and originates from a geologically and physically protected underground water source.	Minerals for Your Body. Drinking mineral water is beneficial to health and well-being. But it is not only the amount of water you drink that is important-what the water contains is even more essential.inerals for Your Body. Drinking mineral water is beneficial to health and well-being. But it is not only the amount of water you drink that is important-what the water contains is even more essential.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 16,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• learning_rate: 8e-05
• num_train_epochs: 1
• warmup_ratio: 0.05
• bf16: True
• dataloader_num_workers: 8
• dataloader_pin_memory: False
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 512
• per_device_eval_batch_size: 512
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 8e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 8
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: False
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	msmarco-co-condenser-dev_cosine_accuracy
-1	-1	-	0.6060
0.0049	2	6.4708	-
0.0099	4	6.3894	-
0.0148	6	6.1023	-
0.0197	8	5.5871	-
0.0246	10	4.5847	-
0.0296	12	3.614	-
0.0345	14	3.117	-
0.0394	16	2.6449	-
0.0443	18	2.0776	-
0.0493	20	1.5595	-
0.0542	22	1.1673	-
0.0591	24	0.9387	-
0.0640	26	0.7596	-
0.0690	28	0.6459	-
0.0739	30	0.5873	-
0.0788	32	0.5577	-
0.0837	34	0.4977	-
0.0887	36	0.4814	-
0.0936	38	0.445	-
0.0985	40	0.4145	-
0.1034	42	0.3985	-
0.1084	44	0.4007	-
0.1133	46	0.3819	-
0.1182	48	0.3581	-
0.1232	50	0.3425	-
0.1281	52	0.3235	-
0.1330	54	0.314	-
0.1379	56	0.3191	-
0.1429	58	0.2999	-
0.1478	60	0.312	-
0.1527	62	0.2948	-
0.1576	64	0.2839	-
0.1626	66	0.2793	-
0.1675	68	0.2853	-
0.1724	70	0.2858	-
0.1773	72	0.2625	-
0.1823	74	0.2804	-
0.1872	76	0.2532	-
0.1921	78	0.2521	-
0.1970	80	0.2501	-
0.2020	82	0.2556	-
0.2069	84	0.2512	-
0.2118	86	0.2371	-
0.2167	88	0.2466	-
0.2217	90	0.231	-
0.2266	92	0.2479	-
0.2315	94	0.2263	-
0.2365	96	0.2352	-
0.2414	98	0.2331	-
0.2463	100	0.2296	-
0.2512	102	0.2152	-
0.2562	104	0.2242	-
0.2611	106	0.2196	-
0.2660	108	0.2118	-
0.2709	110	0.2125	-
0.2759	112	0.2108	-
0.2808	114	0.2097	-
0.2857	116	0.2173	-
0.2906	118	0.2121	-
0.2956	120	0.2173	-
0.3005	122	0.2199	-
0.3054	124	0.202	-
0.3103	126	0.212	-
0.3153	128	0.2047	-
0.3202	130	0.2005	-
0.3251	132	0.2093	-
0.3300	134	0.1948	-
0.3350	136	0.2012	-
0.3399	138	0.2019	-
0.3448	140	0.1906	-
0.3498	142	0.1926	-
0.3547	144	0.1946	-
0.3596	146	0.1937	-
0.3645	148	0.191	-
0.3695	150	0.181	-
0.3744	152	0.1914	-
0.3793	154	0.1844	-
0.3842	156	0.1935	-
0.3892	158	0.1889	-
0.3941	160	0.1939	-
0.3990	162	0.1876	-
0.4039	164	0.178	-
0.4089	166	0.1865	-
0.4138	168	0.1784	-
0.4187	170	0.1728	-
0.4236	172	0.1768	-
0.4286	174	0.183	-
0.4335	176	0.1787	-
0.4384	178	0.1704	-
0.4433	180	0.1754	-
0.4483	182	0.172	-
0.4532	184	0.1654	-
0.4581	186	0.1783	-
0.4631	188	0.1675	-
0.4680	190	0.1713	-
0.4729	192	0.1727	-
0.4778	194	0.1814	-
0.4828	196	0.1632	-
0.4877	198	0.1662	-
0.4926	200	0.1638	-
0.4975	202	0.1623	-
0.5025	204	0.1669	-
0.5074	206	0.1642	-
0.5123	208	0.1747	-
0.5172	210	0.1718	-
0.5222	212	0.1671	-
0.5271	214	0.1587	-
0.5320	216	0.1598	-
0.5369	218	0.1711	-
0.5419	220	0.1635	-
0.5468	222	0.1538	-
0.5517	224	0.148	-
0.5567	226	0.1597	-
0.5616	228	0.1616	-
0.5665	230	0.1577	-
0.5714	232	0.1604	-
0.5764	234	0.1597	-
0.5813	236	0.1627	-
0.5862	238	0.1605	-
0.5911	240	0.1598	-
0.5961	242	0.1577	-
0.6010	244	0.1615	-
0.6059	246	0.1646	-
0.6108	248	0.1512	-
0.6158	250	0.1549	-
0.6207	252	0.154	-
0.6256	254	0.147	-
0.6305	256	0.1457	-
0.6355	258	0.1573	-
0.6404	260	0.1486	-
0.6453	262	0.1454	-
0.6502	264	0.1531	-
0.6552	266	0.1507	-
0.6601	268	0.1493	-
0.6650	270	0.1525	-
0.6700	272	0.1381	-
0.6749	274	0.147	-
0.6798	276	0.145	-
0.6847	278	0.1411	-
0.6897	280	0.1496	-
0.6946	282	0.1497	-
0.6995	284	0.1388	-
0.7044	286	0.1485	-
0.7094	288	0.1487	-
0.7143	290	0.1426	-
0.7192	292	0.1464	-
0.7241	294	0.1446	-
0.7291	296	0.145	-
0.7340	298	0.1429	-
0.7389	300	0.149	-
0.7438	302	0.139	-
0.7488	304	0.1442	-
0.7537	306	0.144	-
0.7586	308	0.1486	-
0.7635	310	0.1367	-
0.7685	312	0.1453	-
0.7734	314	0.1501	-
0.7783	316	0.1434	-
0.7833	318	0.1451	-
0.7882	320	0.1438	-
0.7931	322	0.1424	-
0.7980	324	0.1373	-
0.8030	326	0.145	-
0.8079	328	0.144	-
0.8128	330	0.1437	-
0.8177	332	0.1439	-
0.8227	334	0.1371	-
0.8276	336	0.1275	-
0.8325	338	0.1497	-
0.8374	340	0.1443	-
0.8424	342	0.1372	-
0.8473	344	0.1328	-
0.8522	346	0.1303	-
0.8571	348	0.1429	-
0.8621	350	0.134	-
0.8670	352	0.1372	-
0.8719	354	0.1424	-
0.8768	356	0.1471	-
0.8818	358	0.1409	-
0.8867	360	0.1361	-
0.8916	362	0.1222	-
0.8966	364	0.1355	-
0.9015	366	0.1308	-
0.9064	368	0.1423	-
0.9113	370	0.1352	-
0.9163	372	0.1368	-
0.9212	374	0.1355	-
0.9261	376	0.1353	-
0.9310	378	0.1368	-
0.9360	380	0.1328	-
0.9409	382	0.1309	-
0.9458	384	0.1278	-
0.9507	386	0.1309	-
0.9557	388	0.1332	-
0.9606	390	0.1317	-
0.9655	392	0.1314	-
0.9704	394	0.1336	-
0.9754	396	0.1405	-
0.9803	398	0.1382	-
0.9852	400	0.139	-
0.9901	402	0.1345	-
0.9951	404	0.138	-
1.0	406	0.1389	-
-1	-1	-	0.9710

Framework Versions

• Python: 3.10.0
• Sentence Transformers: 5.2.2
• Transformers: 4.56.2
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}