wylupek/au-blog-rag-embedder

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Compare and contrast the build times for the project under different approaches, including non-virtualized architecture, virtualization with no cache, and virtualization with several improvements. How did the build times change with each iteration?',
    "After so many improvements we've noticed that the build started to be even faster when compared to the non-virtualized architecture times\n\nApproximate build times for the same project:\n\n- 1st approach with no virtualization ~ 10 minutes\n- 1st iteration of virtualization with no cache and other improvements ~ 30 minutes\n- virtualization with several improvements ~ 4-5 minutes\n\nAs you can see, we've achieved a very scalable CI system that seems to be ideal...\n\n## Current issues",
    'Pro-tips first:\n\n- note, that in order to use FireStick in your app, you need to download AmazonFling and WhisperPlay jars from here and include them in your project.\n- while launching the app after integrating FireStick you can get a similar crash:java.lang.NoClassDefFoundError: Failed resolution of: Lorg/apache/http/conn/util/InetAddressUtils;\n      at com.amazon.whisperlink.android.util.RouteUtil.createRoute(RouteUtil.java:78)\n      at com.amazon.whisperlink.android.util.RouteUtil.createRoute(RouteUtil.java:51)\n            ...\nCaused by: java.lang.ClassNotFoundException: Didn\'t find class "org.apache.http.conn.util.InetAddressUtils"\n\nIt is caused by lack of apache networking library in android runtime, since Android 6. To fix it, add the following code to your AndroidManifest.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.8421
cosine_accuracy@3	0.9498
cosine_accuracy@5	0.9761
cosine_accuracy@10	0.9928
cosine_precision@1	0.8421
cosine_precision@3	0.3166
cosine_precision@5	0.1952
cosine_precision@10	0.0993
cosine_recall@1	0.8421
cosine_recall@3	0.9498
cosine_recall@5	0.9761
cosine_recall@10	0.9928
cosine_ndcg@10	0.9219
cosine_mrr@10	0.8987
cosine_map@100	0.899

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,664 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 13 tokens mean: 30.91 tokens max: 88 tokens	min: 8 tokens mean: 175.1 tokens max: 256 tokens

• Samples:

  sentence_0	sentence_1
  How does the 'code-coverage' job in the workflow ensure that it has access to the necessary test results before running the code coverage analysis?	``` code-coverage: name: Merged code coverage runs-on: ubuntu-20.04 permissions: pull-requests: write needs: - unit-tests steps: - name: Checkout uses: actions/checkout@v4 - name: Download tests results for both jobs uses: actions/download-artifact@v4 with: name: test-results-unit name: test-results - name: Run code coverage run: ./gradlew codeCoverage - name: Store HTML coverage report uses: actions/upload-artifact@v4 with: name: coverage-report path:
  Describe the purpose and functionality of the 'Download tests results for both jobs' and 'Store HTML coverage report' steps in the workflow.	``` code-coverage: name: Merged code coverage runs-on: ubuntu-20.04 permissions: pull-requests: write needs: - unit-tests steps: - name: Checkout uses: actions/checkout@v4 - name: Download tests results for both jobs uses: actions/download-artifact@v4 with: name: test-results-unit name: test-results - name: Run code coverage run: ./gradlew codeCoverage - name: Store HTML coverage report uses: actions/upload-artifact@v4 with: name: coverage-report path:
  Explain the purpose of the Payment.Worker module in the given code snippet. How does it utilize the GenServer behavior and what is the significance of the @interval attribute?	``` defmodule Payment.Worker do use GenServer @interval 10 * 6000 def start_link() do ... end def init() do Process.send_after(self(), :work, @interval) {:ok, %{}} end def handle_info(:work, state) do Repo.transaction(fn -> Documents.Accepted.fetch()

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• num_train_epochs: 1
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 10
• per_device_eval_batch_size: 10
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	cosine_ndcg@10
0.2994	50	0.9175
0.5988	100	0.9152
0.8982	150	0.9211
1.0	167	0.9219

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu124
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}