End of training

88b4661 verified about 1 month ago

29.3 kB

language:
  - en
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:477792
  - loss:CachedMultipleNegativesRankingLoss
base_model: answerdotai/ModernBERT-base
widget:
  - source_sentence: tachyphylaxis definition
    sentences:
      - >-
        1 In some areas plumbers charge $45 -$75 an hour; in other regions the
        hourly rate can be $75 -$150. 2  Most plumbers charge a two-hour minimum
        or a service call fee of $75 -$150, and some plumbers bill a flat fee
        per job instead of an hourly rate.3  Either away, exact costs will
        depend on the complexity and type of work done. Plumbers' rates vary
        significantly by location. 2  In some areas plumbers charge $45 -$75 an
        hour; in other regions the hourly rate can be $75 -$150.
      - "Medical Definition of tachyphylaxis. plural. tachyphylaxes. \\-Ë\x8CsÄ\x93z\\play. : diminished response to later increments in a sequence of applications of a physiologically active substance (as the diminished pressor response that follows repeated injections of renin)"
      - >-
        Quick Answer. Injury to the phrenic nerve can paralyze the diaphragm and
        have a serious impact on the regulation of breathing, such as difficulty
        during inhalation, according to the UCLA Division of Plastic &
        Reconstructive Surgery. The phrenic nerve is responsible for the
        function of the diaphragm. Continue Reading.
  - source_sentence: where is st malo beach
    sentences:
      - >-
        Nausea is a sensation of discomfort in the upper abdomen, accompanied by
        an urge to vomit. Also known of as qualm, nausea may be a side effect
        associated with several medications or a symptom of disease or disorder.
        Sometimes large, fatty or sugary meals may also lead to a feeling of
        nausea. Nausea is a sensation of discomfort in the upper abdomen,
        accompanied by an urge to vomit. Also known of as qualm, nausea may be a
        side effect associated with several medications or a symptom of disease
        or disorder.
      - >-
        Location of Pennsylvania in the United States. Folsom is a
        census-designated place (CDP) in Delaware County, Pennsylvania, United
        States. It is part of Ridley Township. The population was 8,323 at the
        2010 census.
      - "Saint Malo Beach Oceanside homes. Developed in the late 1920â\x80\x99s, the community of Saint Malo Beach is one of the highlights of beautiful Carlsbad, CA â\x80\x93 and one of its most secluded hideaways."
  - source_sentence: who invented cotton candy dr pepper and electric chair
    sentences:
      - "Skill positions in football are the positions that are most responsible for causing or preventing points from being scored. The skill positions are: Skill positions are often contrasted with linemen â\x80\x93 players who line up along the line of scrimmage. Skill position players are generally smaller than linemen, but they must also be faster and have other talents (such as the ability to throw or catch the ball, cover an opposing receiver, or to dodge opponents) that rely more on finesse than on brute force."
      - >-
        A WBS Dictionary is merely a supporting document, which provides the
        definitions for each component contained in the Work Breakdown
        Structure. This type of dictionary is often recommended as a reference
        resource material for task-oriented projects comprising several work
        phases.
      - >-
        Cotton Candy (1897): Cotton Candy was invented in 1897 by the American
        inventors William Morrison and John C. Wharton. Cotton Gin (1793): The
        Cotton Gin was invented in 1793 by the American inventor Eli Whitney
        during the Industrial Revolution.
  - source_sentence: what county is boston, ma
    sentences:
      - >-
        There are two kinds of clauses: independent and dependent clauses. Most
        simply, an independent clause can form a complete sentence on its own
        and a dependent clause cannot (at least, not by itself). Think of it
        this way: an independent clause is like a cup of coffee, and a dependent
        clause is like a caffeine lover. Caffeine lovers are dependent on
        coffee, so the two can be joined (quite happily) to form a cohesive
        unit. Similarly, two cups of coffee, or two independent clauses, can be
        combined.
      - >-
        Boston is in the county of Suffolk in Massachusetts. The population is
        about 722,023, and Boston is the largest city.
      - >-
        Pre-diabetes is diagnosed by any one of the following: 1  A fasting
        blood glucose in between 100-125 mg/dL. 2  An A1c between 5.7 - 6.4
        percent.  Any value between 140 mg/dL and 199 mg/dL during a two-hour
        75g oral glucose tolerance test.
  - source_sentence: product key windows 8.1 how to find
    sentences:
      - >-
        If Windows 8.1 came preinstalled on your computer, your Windows 8.1
        product key should be on a sticker on your computer or with your
        documentation. The Windows 8.1 product key is a series of 25 letters and
        numbers and should look like this: xxxxx-xxxxx-xxxxx-xxxxx-xxxxx.
      - >-
        Al Gore not divorced from wife Tipper, confirms relationship with
        longtime girlfriend. 1  Pucker up! Al Gore planted a wet one on wife
        Tipper in 2000, during his presidential campaign. Ten years later the
        couple separated after 40 years of marriage.
      - >-
        springer spaniel. n. 1. (Breeds) either of two breeds of large
        quick-moving spaniels bred to spring game, having a slightly domed head
        and ears of medium length. The English springer spaniel is the larger
        and can be of various colours; the Welsh springer spaniel is always a
        rich red and white. n.
datasets:
  - sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: SentenceTransformer based on answerdotai/ModernBERT-base
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: eval
          type: eval
        metrics:
          - type: cosine_accuracy@1
            value: 0.7949656022587187
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9252395912037221
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.9530759136278681
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9735157275221696
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7949656022587187
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.3084131970679073
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19061518272557365
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09735157275221698
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.7949656022587187
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9252395912037221
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.9530759136278681
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9735157275221696
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8908733180956838
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8636181159543801
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.864765622765834
            name: Cosine Map@100

SentenceTransformer based on answerdotai/ModernBERT-base

This is a sentence-transformers model finetuned from answerdotai/ModernBERT-base on the msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: answerdotai/ModernBERT-base
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1
Language: en

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'OptimizedModule'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("modernbert-msmarco")
# Run inference
queries = [
    "product key windows 8.1 how to find",
]
documents = [
    'If Windows 8.1 came preinstalled on your computer, your Windows 8.1 product key should be on a sticker on your computer or with your documentation. The Windows 8.1 product key is a series of 25 letters and numbers and should look like this: xxxxx-xxxxx-xxxxx-xxxxx-xxxxx.',
    'springer spaniel. n. 1. (Breeds) either of two breeds of large quick-moving spaniels bred to spring game, having a slightly domed head and ears of medium length. The English springer spaniel is the larger and can be of various colours; the Welsh springer spaniel is always a rich red and white. n.',
    'Al Gore not divorced from wife Tipper, confirms relationship with longtime girlfriend. 1  Pucker up! Al Gore planted a wet one on wife Tipper in 2000, during his presidential campaign. Ten years later the couple separated after 40 years of marriage.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.8319, -0.0147, -0.0184]])

Evaluation

Metrics

Information Retrieval

Dataset: eval
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.795
cosine_accuracy@3	0.9252
cosine_accuracy@5	0.9531
cosine_accuracy@10	0.9735
cosine_precision@1	0.795
cosine_precision@3	0.3084
cosine_precision@5	0.1906
cosine_precision@10	0.0974
cosine_recall@1	0.795
cosine_recall@3	0.9252
cosine_recall@5	0.9531
cosine_recall@10	0.9735
cosine_ndcg@10	0.8909
cosine_mrr@10	0.8636
cosine_map@100	0.8648

Training Details

Training Dataset

msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1

Dataset: msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 at 84ed2d3
Size: 477,792 training samples
Columns: query and positive
Approximate statistics based on the first 1000 samples:
query positive
type string string
details
min: 4 tokens
mean: 9.31 tokens
max: 24 tokens

min: 17 tokens
mean: 81.72 tokens
max: 205 tokens

	query	positive
type	string	string
details	min: 4 tokens mean: 9.31 tokens max: 24 tokens	min: 17 tokens mean: 81.72 tokens max: 205 tokens

Samples:

query	positive
`what is the farthest distance in the universe`	`Depends on what you mean by seeing. The particle horizon is just the furthest. distance light could have traveled to us since the universe began. That is 93 billion. light years in diameter, or 47 billion light years in any direction, but we can't actually. see anything at that distance.`
`what county is laurel ms in`	Laurel, MS. Online Offers. Laurel is a city located in Jones County in Mississippi, a state of the United States of America. As of the 2000 census, the city had a total population of 18,393 although a significant population increase has been reported following Hurricane Katrina. Located in southeast Mississippi, southeast of Jackson on Tallahala Creek, Laurel was founded in 1882 as a lumber town. An American Indian reservation is located in nearby Sandersville. Laurel is the principal city of the Laurel Micropolitan Statistical Area.
`how to use a beadloom`	How to string your bead loom. To string a loom, attach your nymo thread to one of the small nails at the end of the loom. Run the thread over the metal bars (located on both ends of the loom) and wrap it around one of the small nails on the other end of your loom.ize 8 seed beads are normally to heavy to be used on a loom. The end result would be beadwork that sags in the middle. Every other slow on the metal bar was skipped to accomodate size 8 seed beads. You will not need to do this with seed beads sizes 10-15 that are the correct size beads to use on a bead loom.

Loss: CachedMultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "mini_batch_size": 64,
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Evaluation Dataset

msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1

Dataset: msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1 at 84ed2d3
Size: 25,147 evaluation samples
Columns: query and positive
Approximate statistics based on the first 1000 samples:
query positive
type string string
details
min: 4 tokens
mean: 9.24 tokens
max: 26 tokens

min: 15 tokens
mean: 81.64 tokens
max: 198 tokens

	query	positive
type	string	string
details	min: 4 tokens mean: 9.24 tokens max: 26 tokens	min: 15 tokens mean: 81.64 tokens max: 198 tokens

Samples:

query	positive
`how long it take for a pimple to burst`	`Start doing warm compress over the pimple, so that it gradually gets drained over 2 to 3 days. Do it gently, without giving yourself much pain. Also, apply mupirocin ointment over it twice a day for 3 to 4 days.Read above in detail about dealing with your infected pimple.tart doing warm compress over the pimple, so that it gradually gets drained over 2 to 3 days. Do it gently, without giving yourself much pain. Also, apply mupirocin ointment over it twice a day for 3 to 4 days.`
`tularosa population`	`The Village of Tularosa had a population of 2,677 as of July 1, 2017. Tularosa ranks in the upper quartile for Population Density when compared to the other cities, towns and Census Designated Places (CDPs) in New Mexico. See peer rankings below. The primary coordinate point for Tularosa is located at latitude 33.075 and longitude -106.0173 in Otero County.`
`do some people have their blood flowing in reverse direction`	`As a result, not enough blood flows through the valve. Some valves can have both stenosis and backflow problems. Atresia occurs if a heart valve lacks an opening for blood to pass through. Some people are born with heart valve disease, while others acquire it later in life.`

Loss: CachedMultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "mini_batch_size": 64,
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 1024
num_train_epochs: 1
learning_rate: 2e-05
warmup_steps: 0.1
bf16: True
eval_strategy: epoch
per_device_eval_batch_size: 1024
push_to_hub: True
hub_model_id: modernbert-msmarco
load_best_model_at_end: True
dataloader_num_workers: 4
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

per_device_train_batch_size: 1024
num_train_epochs: 1
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 0.1
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: trackio
eval_strategy: epoch
per_device_eval_batch_size: 1024
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: True
hub_private_repo: None
hub_model_id: modernbert-msmarco
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: True
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: False
dataloader_num_workers: 4
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_backend: None
ddp_timeout: 1800
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	eval_cosine_ndcg@10
0.1071	50	4.1149	-	-
0.2141	100	0.5296	-	-
0.3212	150	0.3000	-	-
0.4283	200	0.2463	-	-
0.5353	250	0.2247	-	-
0.6424	300	0.2032	-	-
0.7495	350	0.1923	-	-
0.8565	400	0.1900	-	-
0.9636	450	0.1888	-	-
1.0	467	-	0.1889	0.8768
0.1071	50	0.1866	-	-
0.2141	100	0.1560	-	-
0.3212	150	0.1455	-	-
0.4283	200	0.1377	-	-
0.5353	250	0.1397	-	-
0.6424	300	0.1351	-	-
0.7495	350	0.1355	-	-
0.8565	400	0.1417	-	-
0.9636	450	0.1468	-	-
1.0	467	-	0.1512	0.8909

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.12
Sentence Transformers: 5.3.0
Transformers: 5.3.0
PyTorch: 2.10.0+cu128
Accelerate: 1.13.0
Datasets: 4.7.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}