README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:127
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
base_model: nomic-ai/modernbert-embed-base
widget:
  - source_sentence: What is the difference between traditional programming and ML?
    sentences:
      - >-
        Over the past few years, the field of ML has advanced rapidly,
        especially in the area of Natural Language Processing (NLP)—the ability
        of machines to understand and generate human language. At the forefront
        of this progress are Large Language Models (LLMs), such as OpenAI’s GPT
        (Generative Pre-trained Transformer), Google’s PaLM, and Meta’s LLaMA
      - >-
        . For example, integrating an LLM into a customer support chatbot might
        involve connecting it to a company’s internal knowledge base, enabling
        it to answer customer questions using accurate, up-to-date information.
      - >-
        A major subset of AI is Machine Learning (ML), which involves algorithms
        that learn from data rather than being explicitly programmed. Instead of
        writing detailed instructions for every task, ML models find patterns in
        large datasets and use these patterns to make predictions or decisions
  - source_sentence: >-
      What is one of the tasks mentioned that involves creating new written
      content?
    sentences:
      - >-
        In summary, AI and ML form the foundation for intelligent automation,
        while LLMs represent a breakthrough in language understanding and
        generation. Integrating these models into real-world systems unlocks
        practical value, turning raw intelligence into tangible solutions
      - >-
        8. Security and Compliance Integrations

        Some organizations are integrating LLMs to detect anomalies in text
        communications (e.g., phishing detection or policy violations). LLMs can
        analyze language usage and flag potentially suspicious behavior more
        flexibly than keyword-based filters.


        Challenges in LLM Integration

        Despite their promise, integrating LLMs comes with challenges:
      - >-
        . These include text generation, summarization, translation, question
        answering, code generation, and more.
  - source_sentence: What is one of the components mentioned alongside AI?
    sentences:
      - >-
        2. Search Engines and Semantic Search

        Traditional keyword-based search systems are being enhanced or replaced
        by semantic search, where LLMs understand the meaning behind queries.
        Instead of just matching words, they interpret intent.
      - >-
        For example, e-commerce websites can deploy LLM-powered assistants to
        help customers find products, track orders, or get personalized
        recommendations—much more effectively than traditional rule-based bots.
      - Introduction to AI, Machine Learning, LLMs, and Their Integration
  - source_sentence: >-
      What is required to provide intelligent features within broader
      applications?
    sentences:
      - >-
        . For instance, a spam filter doesn’t just block emails with specific
        keywords—it learns from thousands of examples what spam typically looks
        like.
      - >-
        The Rise of LLM Integrations

        While LLMs are powerful on their own, their true potential is unlocked
        through integration—connecting these models with other software,
        services, or systems to provide intelligent features within broader
        applications.


        Here are some key ways LLMs are being integrated into the digital world:
      - >-
        For instance, in a document management system, a user might type
        "policies about sick leave", and the system—integrated with an LLM—could
        retrieve documents discussing "medical leave", "employee absence", and
        "illness policies", even if those exact words weren’t used.
  - source_sentence: What type of dialogues can LLMs simulate?
    sentences:
      - >-
        Companies are also experimenting with Retrieval-Augmented Generation
        (RAG)—a technique where LLMs are paired with document databases (e.g.,
        vector stores like Supabase, Pinecone, or Weaviate) to answer questions
        with enterprise-specific knowledge.
      - >-
        . For example, integrating an LLM into a customer support chatbot might
        involve connecting it to a company’s internal knowledge base, enabling
        it to answer customer questions using accurate, up-to-date information.
      - >-
        5. Education and Learning Platforms

        Educational tools like Khanmigo (from Khan Academy) and other tutoring
        platforms are leveraging LLMs to provide real-time help to students.
        LLMs can break down complex topics, provide feedback on writing, and
        simulate Socratic-style dialogues.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
model-index:
  - name: Fine-tuned with [QuicKB](https://github.com/ALucek/QuicKB)
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 768
          type: dim_768
        metrics:
          - type: cosine_accuracy@1
            value: 0.6666666666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 1
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6666666666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2666666666666667
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.20000000000000007
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.10000000000000003
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6666666666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 1
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8310827786456928
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7766666666666667
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7766666666666667
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 512
          type: dim_512
        metrics:
          - type: cosine_accuracy@1
            value: 0.6666666666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8666666666666667
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6666666666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2666666666666667
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.17333333333333337
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.10000000000000003
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6666666666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8666666666666667
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8203966331432972
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7651851851851852
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7651851851851852
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 256
          type: dim_256
        metrics:
          - type: cosine_accuracy@1
            value: 0.6666666666666666
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.8666666666666667
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 1
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.6666666666666666
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.28888888888888886
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.20000000000000007
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.10000000000000003
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.6666666666666666
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.8666666666666667
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 1
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.8357043414408
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7822222222222223
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.7822222222222223
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 128
          type: dim_128
        metrics:
          - type: cosine_accuracy@1
            value: 0.5333333333333333
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.7333333333333333
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.9333333333333333
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.5333333333333333
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.2444444444444445
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.16000000000000003
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09333333333333335
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.5333333333333333
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.7333333333333333
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.9333333333333333
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7203966331432973
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.6540740740740741
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.6592022792022793
            name: Cosine Map@100
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: dim 64
          type: dim_64
        metrics:
          - type: cosine_accuracy@1
            value: 0.4666666666666667
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.6666666666666666
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 0.8
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8666666666666667
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.4666666666666667
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.22222222222222224
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.16000000000000003
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.08666666666666668
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.4666666666666667
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.6666666666666666
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.8
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.8666666666666667
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.6507228370099043
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.5822222222222223
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.58890559732665
            name: Cosine Map@100

Fine-tuned with QuicKB

This is a sentence-transformers model finetuned from nomic-ai/modernbert-embed-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nomic-ai/modernbert-embed-base
• Maximum Sequence Length: 1024 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Nuf-hugginface/modernbert-embed-quickb")
# Run inference
sentences = [
    'What type of dialogues can LLMs simulate?',
    '5. Education and Learning Platforms\nEducational tools like Khanmigo (from Khan Academy) and other tutoring platforms are leveraging LLMs to provide real-time help to students. LLMs can break down complex topics, provide feedback on writing, and simulate Socratic-style dialogues.',
    '. For example, integrating an LLM into a customer support chatbot might involve connecting it to a company’s internal knowledge base, enabling it to answer customer questions using accurate, up-to-date information.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: dim_768, dim_512, dim_256, dim_128 and dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	dim_768	dim_512	dim_256	dim_128	dim_64
cosine_accuracy@1	0.6667	0.6667	0.6667	0.5333	0.4667
cosine_accuracy@3	0.8	0.8	0.8667	0.7333	0.6667
cosine_accuracy@5	1.0	0.8667	1.0	0.8	0.8
cosine_accuracy@10	1.0	1.0	1.0	0.9333	0.8667
cosine_precision@1	0.6667	0.6667	0.6667	0.5333	0.4667
cosine_precision@3	0.2667	0.2667	0.2889	0.2444	0.2222
cosine_precision@5	0.2	0.1733	0.2	0.16	0.16
cosine_precision@10	0.1	0.1	0.1	0.0933	0.0867
cosine_recall@1	0.6667	0.6667	0.6667	0.5333	0.4667
cosine_recall@3	0.8	0.8	0.8667	0.7333	0.6667
cosine_recall@5	1.0	0.8667	1.0	0.8	0.8
cosine_recall@10	1.0	1.0	1.0	0.9333	0.8667
cosine_ndcg@10	0.8311	0.8204	0.8357	0.7204	0.6507
cosine_mrr@10	0.7767	0.7652	0.7822	0.6541	0.5822
cosine_map@100	0.7767	0.7652	0.7822	0.6592	0.5889

Training Details

Training Dataset

Unnamed Dataset

• Size: 127 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 127 samples:

  	anchor	positive
  type	string	string
  details	min: 8 tokens mean: 13.28 tokens max: 25 tokens	min: 13 tokens mean: 53.34 tokens max: 86 tokens

• Samples:

  anchor	positive
  What task mentioned is related to providing answers to inquiries?	. These include text generation, summarization, translation, question answering, code generation, and more.
  What do LLMs learn to work effectively?	LLMs work by learning statistical relationships between words and phrases, allowing them to predict and generate language that feels natural. The power of these models lies not only in their size but also in the diversity of tasks they can perform with little to no task-specific training
  In which industries is the generalization ability considered useful?	. This generalization ability makes them incredibly useful across industries—from customer service and education to software development and healthcare.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 4
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• tf32: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
1.0	4	-	0.7790	0.7120	0.7474	0.6321	0.5684
2.0	8	-	0.8275	0.7966	0.8091	0.6904	0.6102
2.5	10	13.4453	-	-	-	-	-
3.0	12	-	0.8311	0.8204	0.8357	0.7178	0.6557
4.0	16	-	0.8311	0.8204	0.8357	0.7204	0.6507

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.6
• Sentence Transformers: 3.4.0
• Transformers: 4.48.1
• PyTorch: 2.5.1+cpu
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}