ronit01/rag_tuned_minilm_mnr_1epoch

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ronit01/rag_tuned_minilm_mnr_1epoch")
# Run inference
sentences = [
    "How do the Stop and Delete IC Ops compare in terms of their effects on a run's state, visibility on the dashboard, resource usage, artifact preservation, and what further IC Ops can be performed on the run afterward?",
    'Delete\n----\n\nThis IC Op earmarks the run to be deleted from the next chunk onward. \nOn the chart, you will see its curves vanish almost immediately. \nYou cannot do any further IC Ops on a deleted run because it will not be visible. \nNote that although a deleted run vanishes from the plots, its model checkpoints are still part of \nthe artifacts of that experiment so that you have post-hoc audibility.\n',
    'Why Not Just Downsample Data?\n------------\n\nAt first glance, one might consider running multi-config comparisons by downsampling data for \nquick estimates, then running promising configs on full data. While common, this approach is \noften misleading and cumbersome.\n\nA single downsample introduces variance from one static snapshot, potentially leading to wrong \nconclusions, especially with overfitting-prone LLMs/DL models. It requires manual checkpoint \nmanagement, adding tedious file work. You also do not get dynamic control (stop, resume, clone-modify),\nor you must reimplement such tricky operations, taking time away from your AI application work.\n\nRapidFire AI takes such practical heuristics to their logical conclusion with shard-based \nadaptive multi-config execution with dynamic experiment control. This offers you maximum power \nand flexibility for AI development without extra DevOps grunt work, i.e., *rapid experimentation*.\n\nThe above said, note that downsampling is *complementary* to rapid experimentation--feel free \ndo both! The adaptive execution can operate on your downsampled dataset all the same.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5535, 0.2718],
#         [0.5535, 1.0000, 0.1378],
#         [0.2718, 0.1378, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 46 training samples

• Columns: sentence_0 and sentence_1

• Approximate statistics based on the first 46 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 11 tokens mean: 30.57 tokens max: 48 tokens	min: 64 tokens mean: 225.52 tokens max: 256 tokens

• Samples:

  sentence_0	sentence_1
  How does RapidFire AI's shard-based adaptive execution engine enable online aggregation of eval metrics with confidence intervals, and what specific mathematical strategies are available for computing those intervals?
  RapidFire AI transforms the status quo by adapting the powerful idea of online aggregation
  from database systems research to LLM evals.
  Our adaptive execution engine, :doc:as described on this page</difference>, automatically
  shards the data and processes multiple configs in parallel, one shard at a time, with
  efficient swapping techniques.

  This means you get running metric estimates with confidence intervals in real time. So, you can confidently stop poor configs earlier, clone better configs on the fly, and perform more informed exploration to reach much better eval metrics in much less time.

  Example: Traditional Batch Evals vs. RapidFire AI

  For instance, suppose you have an evals set with 400 queries. You decide to compare, say, 4 RAG configs in one go with RapidFire AI with number of shards set to 8. The illustration below contrasts traditional batch evals vs. RapidFire AI's approach for a simple eval metric.

  .. list-table:: :widths: 50 50 :clas... | | How do the Stop and Delete IC Ops compare in terms of their effects on a run's state, visibility on the dashboard, resource usage, artifact preservation, and what further IC Ops can be performed on the run afterward? | Stop
----

This IC Op earmarks a run to be stopped at the end of its current chunk.
It will still be alive but it will not use any GPU resources from the next chunk.
You will still see its minibatch-level plots advancing for the current chunk.
You cannot stop an already stopped or deleted run.


.. raw:: html

style="cursor: zoom-in; max-width: 100%;" onclick="this.requestFullscreen()">

style="cursor: zoom-in; max-width: 100%;" onclick="this.requestFullscreen()">
| | How does the GSM8K tutorial configure its few-shot prompting pipeline, and how many total config combinations does it produce? | This use case notebook features an hybrid workflow spanning a self-hosted open LLM for embeddings and an Open AI call for generation.

  Task, Dataset, and Prompt

  This tutorial shows few-shot prompting as part of context engineering for solving grade school math word problems.

  It uses the "GSM8K" dataset; see its details here <https://huggingface.co/datasets/openai/gsm8k>__. The dataset contains grade school math word problems requiring multi-step reasoning.

  The prompt format includes system instructions defining the assistant as a math problem solver, semantically selected few-shot examples, and the target question to solve.

  Model, Few-Shot Selection, and Configuration Knobs

  We compare 2 generator models via OpenAI API: gpt-5-mini and gpt-4o.

  There are 2 different reasoning effort levels for the first model only: medium and high.

  The few-shot prompting pipeline uses:

  • Example Selection: Semantic similarity-based selection using sentence-transformers/... |

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 1
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Time

• Training: 2.3 seconds

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}