SentenceTransformer based on nomic-ai/modernbert-embed-base

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base on the ssf-train-valid-combi-v1v2v3 dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: nomic-ai/modernbert-embed-base
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- ssf-train-valid-combi-v1v2v3

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("frankwong2001/4_modernbert-embed-base")
# Run inference
sentences = [
    'The Assistant Pastry Cook/Assistant Baker/Kitchen Assistant is responsible for the production of pastry and baked goods. He/She prepares the baking equipment and ingredients, and applies finishing touches in post-production of pastries and baked goods. He follows hygiene, safety and other standards, and carries out food and beverage operational tasks. He may suggest areas for continuous improvement within his own workstation. Attentive and meticulous, he possesses good time management skills and is able to multi-task, while performing physical tasks in a high-volume production environment. He is able to work under high temperatures, and in a flexible schedule, including weekends, evenings, and public holidays.',
    'The Assistant Pastry Cook/Assistant Baker/Kitchen Assistant is tasked with producing a variety of pastries and baked goods. He/She prepares the necessary baking equipment and ingredients, and adds finishing touches during the post-production phase of the items. He adheres to hygiene, safety, and other relevant standards, while executing food and beverage operational duties. He may also identify opportunities for continuous improvement in his workstation. Detail-oriented and diligent, he demonstrates strong time management skills and the ability to multi-task in a fast-paced production setting. He is capable of working in high temperatures and maintains a flexible schedule, which may include weekends, evenings, and public holidays.',
    'The Assistant Chef/Line Cook/Kitchen Staff is responsible for the preparation of savory dishes and entrees. He/She sets up the cooking appliances and ingredients, and applies garnishes during the serving of dishes. He follows cleanliness, safety, and other protocols, and performs kitchen and dining operational duties. He may recommend changes for better efficiency within his own cooking area. Attentive and thorough, he exhibits strong organizational skills and is adept at prioritizing tasks, while executing routine duties in a low-volume service environment. He is able to work in cold conditions, and adheres to a rigid schedule, which excludes weekends, evenings, and public holidays.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9323, 0.2374],
#         [0.9323, 1.0000, 0.2841],
#         [0.2374, 0.2841, 1.0000]])

Training Details

Training Dataset

ssf-train-valid-combi-v1v2v3

Dataset: ssf-train-valid-combi-v1v2v3 at cf63f9b
Size: 10,556 training samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative
type	string	string	string
details	min: 63 tokens mean: 168.29 tokens max: 355 tokens	min: 66 tokens mean: 161.05 tokens max: 317 tokens	min: 35 tokens mean: 175.65 tokens max: 1455 tokens

Samples:

anchor	positive	negative
Lighting Designers are responsible for crafting lighting designs in line with the creative vision of the production. They are responsible for designing all aspects of lighting; from creating plans and specifying equipment placement to cue development, including brightness, colour and transitions. They are also responsible for all paperwork related to lighting, such as hook-ups, schedules and colour lists. They create lighting plots specifying the placement and configuration of all instruments and oversee lighting during production runs. Lighting Designers lead the lighting team and coordinate the development, installation and operation of the lighting design and any other special electrical effects. They decide on the lighting equipment to use from existing inventory or on the lighting rental package for venues with cold rig. They work in tandem with the creative leadership, other production designers and the lighting team to ensure the lighting complements all creative elements of the...	The Lighting Designer is tasked with developing innovative lighting concepts that align with the artistic direction of the performance. This role involves designing every detail of the lighting setup, from drafting plans and determining equipment locations to developing cues that encompass brightness, color, and transitions. Additionally, the Lighting Designer manages all documentation related to lighting, including connection details, schedules, and color charts. They create comprehensive lighting plots that outline the arrangement and configuration of all lighting fixtures and supervise the lighting during live performances. Leading the lighting team, the Lighting Designer coordinates the design, setup, and operation of the lighting scheme, as well as any unique electrical effects. They make informed decisions regarding the lighting gear to be utilized from the available inventory or select rental packages for venues with limited equipment. Collaborating closely with the creative tea...	The Lighting Technician is responsible for maintaining the sound systems in line with the overall production requirements. They handle all aspects of audio setup, from developing sound plans and specifying equipment placement to cue development, including volume, effects, and transitions. They also manage all paperwork related to sound, such as connection diagrams, schedules, and effect lists. The Lighting Technician creates sound plots specifying the placement and configuration of all audio instruments and oversees sound during rehearsal runs. They lead the sound team and coordinate the development, installation, and operation of the audio design and any other special sound effects. They decide on the audio equipment to use from existing inventory or on the sound rental package for venues with limited resources. They work closely with the technical team to ensure the sound complements all technical elements of the production.
The Associate Director (Facilities Management) is responsible for driving strategies to improve facility operations. He/She builds strategic relationships with stakeholders and drives service excellence. He formulates organisational Workplace Safety and Health (WSH) practices as well as the green building strategies to fulfil environmental sustainability regulations. He is in charge of approving tender specifications, awarding works to selected bidders and endorsing contracts. He oversees the teams' development and recruitment and is responsible for the departments' financial planning and risk management. He is a subject matter expert and possesses excellent negotiation and people management skills. He is able to influence and communicate effectively with internal and external stakeholders.	The Associate Director (Facilities Management) plays a pivotal role in enhancing the efficiency of facility operations. This position involves cultivating strong partnerships with key stakeholders and championing exceptional service delivery. The individual is tasked with developing organizational practices for Workplace Safety and Health (WSH) alongside implementing sustainable green building initiatives to meet environmental regulations. Responsibilities also include approving tender proposals, selecting contractors for projects, and endorsing contractual agreements. Additionally, the Associate Director oversees team growth and recruitment efforts, manages the department's financial strategies, and addresses risk management. A recognized expert in the field, the individual demonstrates outstanding negotiation and interpersonal skills, effectively influencing and engaging with both internal and external parties.	The Associate Director (Event Management) coordinates various events and activities, focusing on logistics and attendee engagement. The Associate Director (Facilities Management) is responsible for overseeing entry-level staff and managing basic office supplies without strategic oversight. The Associate Director (Quality Assurance) ensures compliance with industry standards in a manufacturing environment, focusing on product testing and regulatory reporting. The Associate Director (Facilities Management) manages operations in a multicultural context, aligning with international regulations and cross-border practices. The Associate Director (Project Management) combines project planning and client relations, tasked with both strategic oversight and detailed operational execution across multiple projects.
The Derivatives Trading Manager/Senior Derivatives Trader assumes responsibilities of quantifiable derivative trading portfolios and their supporting activities. He/She plans derivative trading activities that support his portfolio objectives and take leads in structured products' deal making. He is expected to develop derivative portfolio strategies to guide positions during various market trends or economic conditions. Armed with strong numerical and business acumen, he possesses a good understanding of market conditions as well.	The Derivatives Trading Manager is responsible for managing quantifiable derivative trading portfolios and their associated activities. He/She plans trading initiatives that align with portfolio objectives and takes the lead in structuring product deals. He is expected to formulate derivative portfolio strategies to navigate positions during diverse market trends or economic circumstances. Equipped with strong numerical skills and business insight, he has a solid grasp of market dynamics as well.	`The Derivatives Trading Assistant manages non-quantifiable trading activities and their unrelated tasks. He/She organizes trading schedules that do not align with portfolio goals and takes a supportive role in unstructured product negotiations. He is expected to assist in developing non-derivative strategies to navigate decisions during unrelated market fluctuations or economic situations. Lacking strong numerical skills, he has minimal understanding of market dynamics as well.`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Evaluation Dataset

ssf-train-valid-combi-v1v2v3

Dataset: ssf-train-valid-combi-v1v2v3 at cf63f9b
Size: 2,639 evaluation samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative
type	string	string	string
details	min: 57 tokens mean: 170.84 tokens max: 352 tokens	min: 55 tokens mean: 163.57 tokens max: 319 tokens	min: 20 tokens mean: 189.74 tokens max: 1282 tokens

Samples:

anchor	positive	negative
The General Manager/Managing Director/Vice President (Aircraft Maintenance) is responsible for defining the long-term strategic direction to grow the business in line with the organisations overall vision, mission and values. He/She promotes strategic aircraft maintenance programmes for business competitiveness and sets direction for leading aerospace maintenance practices in the organisation. He represents the organisation with customers, investors, and business partners, and holds responsibility for promoting organisational compliance with airworthiness and legislative requirements, fostering a culture of workplace safety and health, and championing leading practices and quality and risk management. He inspires the organisation towards achieving business goals by striving for continuous improvement, driving digital innovation and evaluating the organisation's approach towards a lean and sustainable enterprise. He demonstrates excellent leadership capabilities and builds strategic par...	The General Manager/Managing Director/Vice President (Aircraft Maintenance) is tasked with shaping the long-term strategic vision to enhance business growth in alignment with the organization's core values and mission. He/She advocates for strategic aircraft maintenance initiatives to ensure business competitiveness and provides direction for leading aerospace maintenance practices within the organization. He represents the organization to customers, investors, and business partners, while also ensuring compliance with airworthiness and legislative standards, promoting a culture of workplace safety and health, and championing best practices in quality and risk management. He motivates the organization to reach its business objectives through continuous improvement, driving digital innovation, and assessing the organization's commitment to a lean and sustainable enterprise. He showcases strong leadership skills and cultivates strategic partnerships with stakeholders to advance business ...	The Software Engineer develops innovative applications and ensures software quality through rigorous testing and debugging processes. He/She collaborates with design teams to create user-friendly interfaces and integrates feedback to enhance functionality and performance. He is responsible for maintaining code repositories and documenting software processes while staying updated with the latest technological advancements. He also participates in team meetings to discuss project progress and brainstorm new ideas for product improvement.
The Associate User Interface Designer performs requirements analysis for the design of user interfaces (UIs) and drafts technical specifications for the design of UIs. He/She assists in the development and programming of intuitive and responsive UIs for each screen or page with which a user interacts. He assists in developing prototypes for UIs, conducts usability testing for validation, and supports the evaluation of the effectiveness of the UI. He prepares reports on UI design performance indicators, proposes, modifications in the design of user interface based on user feedback, as well as solutions to address design issues. He works in a team and is familiar with programming languages used by the organisation to design and develop UIs. He is familiar with graphic designing tools, and is also knowledgeable of Universal Principles of Design as well as commonly used design methods. The Associate UI Designer adopts a broad perspective to user interface design concepts, and is open to ex...	The Associate User Interface Designer is responsible for conducting requirements analysis to create user interfaces (UIs) and drafting technical specifications for UI design. He/She contributes to the development and programming of intuitive and responsive UIs for each user interaction screen or page. He assists in creating prototypes for UIs, performs usability testing for validation, and supports the assessment of UI effectiveness. He prepares reports on UI design performance metrics, suggests modifications based on user feedback, and offers solutions to design challenges. He collaborates within a team and is proficient in the programming languages utilized by the organization for UI design and development. He is knowledgeable in graphic design tools and well-versed in Universal Principles of Design and common design methodologies. The Associate UI Designer takes a comprehensive view of user interface design concepts and is eager to explore innovative options in the development of so...	The Associate Software Engineer conducts requirements analysis for the development of software applications and drafts technical specifications for application functionality. He/She assists in the coding and debugging of efficient and scalable applications for various platforms. He aids in creating application prototypes, performs functionality testing for validation, and supports the analysis of application performance metrics. He prepares reports on application development indicators, suggests enhancements based on user input, and provides solutions to functionality issues. He works within a team and is familiar with programming languages employed by the organization for software development. He is experienced with database management tools and knowledgeable of Software Development Life Cycle methodologies as well as commonly used programming practices. The Associate Software Engineer adopts a narrow focus on application development concepts and is hesitant to explore new approaches ...
The Senior Project Engineer is responsible for executing project management plans from start to finish, to ensure project completions on time, and within budget. He/She typically comes from an engineering background with work experience in production and/or design, and is able to develop project schedules, budgets and manage project staff and subcontractors. He has good communication and negotiation skills for engaging internal and external parties to secure specialised resources and contributions for projects, and managing ongoing relationships with sub-contractors. He oversees sub-contractors schedules, performance, and payments, and has the responsibility to reschedule and coordinate work to ensure compliance with applicable project schedules.	The Senior Project Engineer plays a pivotal role in implementing comprehensive project management strategies from inception to completion, ensuring that projects are delivered on time and within financial constraints. This position typically requires a background in engineering, along with relevant experience in production and design. The Senior Project Engineer is adept at creating project timelines and budgets while effectively managing project teams and subcontractors. Strong communication and negotiation skills are essential for collaborating with both internal and external stakeholders to secure specialized resources and maintain productive relationships with subcontractors. Additionally, this role involves monitoring subcontractor schedules, performance metrics, and payment processes, along with the responsibility to adjust and coordinate work to align with project timelines.	The Senior Project Engineer is tasked with overseeing the financial audits of various departments, ensuring compliance with internal policies and external regulations. The Senior Project Engineer is responsible for mentoring junior engineers while managing large-scale engineering projects, requiring at least ten years of experience in a senior leadership role. The Senior Project Engineer focuses on compliance checks within the healthcare sector, utilizing analytical skills to assess regulatory frameworks and implement necessary changes. The Senior Project Engineer is involved in developing marketing strategies for consumer products, requiring expertise in market analysis and brand management across international markets. The Senior Project Engineer combines responsibilities of a project manager and a quality assurance officer, overseeing project execution while also conducting product testing and compliance evaluations.

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 16
gradient_accumulation_steps: 8
learning_rate: 2e-05
num_train_epochs: 5
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: True
tf32: False
load_best_model_at_end: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 8
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 5
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: False
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss
1.0	83	0.0317	0.0009
2.0	166	0.0009	0.0006
3.0	249	0.0006	0.0006
4.0	332	0.0006	0.0005
5.0	415	0.0005	0.0005

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.12.11
Sentence Transformers: 5.1.0
Transformers: 4.55.0
PyTorch: 2.8.0+cu128
Accelerate: 1.10.0
Datasets: 4.0.0
Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}