ronit01/golden_rag_tuned_minilm_100

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ronit01/golden_rag_tuned_minilm_100")
# Run inference
sentences = [
    'What are all the Experiment class methods (experiment ops) provided by RapidFire AI, and what does each one do?',
    '.. py:class:: RFLangChainRagSpec\n\n  .. py:method:: __init__(document_loader: BaseLoader = None, text_splitter: TextSplitter = None, embedding_cfg: dict[str, Any] = None, vector_store_cfg: dict[str, Any] = None, retriever: BaseRetriever = None, search_cfg: dict[str, Any] = None, reranker_cfg: dict[str, Any] = None, enable_gpu_search: bool = False, document_template: Callable[[Document], str] = None)\n\n    Initialize the RAG specification with document loading, chunking, embedding, indexing, retrieval, and reranking configurations.\n\n    :param document_loader: The loader for source documents from various sources (files, directories, databases, etc.). Must be a LangChain BaseLoader implementation.\n    :type document_loader: BaseLoader, optional\n\n    :param text_splitter: The text splitter for chunking documents for RAG purposes. Controls chunk size, overlap, and splitting strategy. Must be a LangChain TextSplitter.\n    :type text_splitter: TextSplitter, optional\n\n    :param embedding_cfg: The embedding class and its kwargs to convert a chunk/query into a vector, provided as a single dictionary. Must include a key :code:`"class"` with the class itself as value, not an instance. Options for the class include :class:`HuggingFaceEmbeddings` and :class:`OpenAIEmbeddings`. The kwargs that follow must contain all parameters needed to initialize the embedding class; required parameters vary by embedding class. For example, :class:`HuggingFaceEmbeddings` needs :code:`model_name`, :code:`model_kwargs` and :code:`device`, while :class:`OpenAIEmbeddings` needs :code:`"model"` and :code:`"api_key"`.\n    :type embedding_cfg: dict[str, Any], optional\n\n    :param vector_store_cfg: The vector store type and args to store and possibly index embedding vectors for retrieval, provided as a single dictionary. \n    \n        - :code:`"type"`: The type of vector store to use. Must be one of :code:`"faiss"`, :code:`"pgvector"`, or :code:`"pinecone"`. Required.\n        - :code:`"batch_size"`: Number of vectors per insert batch. Applies to all 3 types of stores. Optional; default is 128.\n\n        The remaining keys are type-specific args as listed below. The vector store operates in one of 3 modes depending on the rest of the RAG spec:\n\n        - **Create mode:** When :code:`document_loader` is provided and no pre-existing index/collection names are specified, a new vector store is *created* and populated from the loaded documents.\n        - **Read mode:** When :code:`document_loader` is absent and pre-existing index/collection names are specified, the vector store is opened in *read-only* mode for retrieval against the existing index.\n        - **Update mode:** When both :code:`document_loader` and pre-existing index/collection names are provided, the existing index/collection is *updated* with the new documents added to it.\n\n        Supported vector store types and their arg keys:\n\n        - **FAISS:** No additional keys. Uses a flat L2 index by default. Set :code:`enable_gpu_search=True` on the constructor to use GPU-accelerated FAISS. Only supports Create mode since it\'s an in-memory store that is not persistent. So, the notion of pre-existing indexes does not apply.\n\n        - **Pinecone:**\n\n          - :code:`"pinecone_api_key"`: Pinecone API key. Optional if the :code:`PINECONE_API_KEY` environment variable is set.\n          - :code:`"index_namespace"`: A 2-tuple of strings (:code:`tuple[str, str]`) with index name and namespace. Required for Read/Update mode and must be a pre-existing index and namespace (NB: namespace can be empty string :code:`""` in Pinecone). N/A for Create mode.\n          - :code:`"spec"`: A :code:`ServerlessSpec` or :code:`PodSpec` instance specifying the Pinecone deployment (e.g., cloud and region). Required for Create mode. N/A for Read/Update mode.\n          - :code:`"metric"`: Distance metric for the index, must be one of :code:`"cosine"`, :code:`"euclidean"`, or :code:`"dotproduct"`. Optional for Create mode; default is :code:`"cosine"`. N/A for Read/Update mode.\n          - :code:`"embedding_cfg"`: Embedding config dict (same format as the top-level :code:`embedding_cfg`). Required for any mode either here or in the top-level config for any mode. If provided here, *this takes precedence* over the top-level embedding config. For Create mode, we recommend providing it in the top-level config unless you want to couple different embedding configs with different vector stores.\n          - :code:`"text_key"`: The metadata field name used to store the original raw text content associated with a vector in Pinecone. Optional; default is :code:`"text"`. Applicable to all modes. This is useful when the Pinecone index was populated by an external tool that stored text under a non-default metadata field name (e.g., :code:`"content"`, :code:`"original_text"`).\n          - :code:`"vector_type"`: Vector type for the index. Accepts a :code:`VectorType` value or string. Optional for Create mode; default is :code:`"dense"`. N/A for Read/Update mode.\n          - :code:`"tags"`: Arbitrary string key-value tags to attach to the index. Optional for Create mode; default is :code:`None`. N/A for Read/Update mode.\n          - :code:`"timeout"`: Timeout in seconds for index operations. Optional for Create mode; default is :code:`None`. N/A for Read/Update mode.\n          - :code:`"deletion_protection"`: Whether deletion protection is enabled. Accepts a :code:`DeletionProtection` value or string. Optional for Create mode; default is :code:`"disabled"`. N/A for Read/Update mode.\n\n          To recap, for all 3 modes :code:`"pinecone_api_key"` is needed either here or as an environment variable; :code:`embedding_cfg` is also required either here or in the top-level config. The :code:`"text_key"` is optional for all modes and defaults to :code:`"text"`. \n          \n          For Create mode, :code:`"spec"` is required but the following are all optional: :code:`"metric"`, :code:`"vector_type"`, :code:`"tags"`, :code:`"timeout"`, and :code:`"deletion_protection"`. Although the argument :code:`"index_namespace"` is inapplicable, internally RapidFire AI creates an index name automatically with prefix "rf-" and an SHA hash per pre-processing worker to avoid naming conflicts; the namespace created is the default empty string.\n          \n          For Read/Update mode, :code:`"index_namespace"` is required and must point to a pre-existing index and namespace. All the other arguments are inapplicable.\n\n        - **Postgres PGVector:**\n\n          - :code:`"connection"`: DB connection string or engine. Required for all modes.\n          - :code:`"collection_name"`: A pre-existing PGVector collection/table name to use for retrieval. Required for Read/Update mode. Inapplicable to Create mode; an SHA-based random name will be generated.\n          - :code:`"embedding_cfg"`: Same explanation as above under Pinecone.\n          - :code:`"pre_delete_collection"`: If :code:`True`, *deletes* the collection if it already exists before writing. **Use with caution.** Optional; default is :code:`False`. Applicable only to Update mode.\n\n        The store is built from the documents provided via :code:`document_loader`. If this entire config is skipped, a default FAISS flat vector store will be created automatically.\n    :type vector_store_cfg: dict[str, Any], optional\n\n    :param retriever: The retriever for chunk retrieval. If not provided, a default FAISS vector store will be created automatically using the specified search configuration below. Must be a LangChain BaseRetriever implementation.\n    :type retriever: BaseRetriever, optional\n\n    :param search_cfg: The search algorithm type and its kwargs to use for retrieval of vectors/chunks, provided as a single dictionary. Must include a key :code:`"type"` with one of the following three options listed as value; default is :code:`"similarity"`.\n\n      * :code:`"similarity"`: Standard cosine similarity search.\n      * :code:`"similarity_score_threshold"`: Similarity search with minimum score threshold (SST).\n      * :code:`"mmr"`: Maximum Marginal Relevance (MMR) search for diversity.\n\n      Additional parameters for search configuration depend on the type; the keys can include the following:\n\n      * :code:`"k"`: Number of documents to retrieve. Default is 5.\n      * :code:`"filter"`: Optional filter criteria function for search results.\n      * :code:`"score_threshold"`: Only for SST. Minimum similarity score threshold. \n      * :code:`"fetch_k"`: Only for MMR. Number of documents to fetch before MMR reranking. Default is 20.\n      * :code:`"lambda_mult"`: Only for MMR. Diversity parameter for MMR balancing relevance vs. diversity. Default is 0.5.\n    :type search_cfg: dict, optional\n\n    :param reranker_cfg: The reranker class and its kwargs for reordering retrieved chunks by relevance, provided as a single dictionary. Must include a key :code:`"class"` with the class itself as value, not an instance. Options include :class:`CrossEncoderReranker` from :code:`langchain.retrievers.document_compressors`. The instantiated reranker is applied to each query\'s results individually. The kwargs that follow must contain all parameters needed to initialize the reranker class; required parameters vary by reranker class. For example, :class:`CrossEncoderReranker` needs :code:`model_name`, :code:`model_kwargs` and :code:`top_n`.\n    :type reranker_cfg: dict[str, Any], optional\n\n    :param enable_gpu_search: If :code:`True`, uses GPU-accelerated FAISS (IndexFlatL2 on GPU) with matrix multiply for exact search. Otherwise uses CPU-based FAISS HNSW index (IndexHNSWFlat) for approximate search. GPU mode requires :code:`faiss-gpu` package and CUDA-compatible GPU. Default is :code:`False`.\n    :type enable_gpu_search: bool, optional\n\n    :param document_template: Optional function to format each retrieved chunk for context injection into prompts. Should accept a single LangChain :class:`Document` object and return a formatted string. Multiple documents are separated by double newlines when serialized. If not provided, the following default template is used:\n \n        .. code-block:: python\n \n            def default_template(doc: Document) -> str:\n                """Default document formatting template."""\n                metadata = "; ".join([f"{k}: {v}" for k, v in doc.metadata.items()])\n                return f"{metadata}:\\n{doc.page_content}"\n \n        You can provide a custom template to control what metadata fields are included and how the content is formatted. For example, to include only a specific metadata field:\n \n        .. code-block:: python\n \n            def sample_template(doc: Document) -> str:\n                doc_source = doc.metadata.get("source", "")\n                return f"Document Source: {doc_source}:\\nContent: {doc.page_content}"\n \n        Or for a dataset like SciFact where documents have a :code:`"title"` metadata field ingested via :code:`metadata_func` in the document loader:\n \n        .. code-block:: python\n \n            def custom_template(doc: Document) -> str:\n                return f"{doc.metadata[\'title\']}: {doc.page_content}"\n \n    :type document_template: Callable[[Document], str], optional',
    'Preprocess Function\n-------------------\n\nMandatory user-provided function to prepare the inputs to be given to the generator model. \nIt is invoked for each batch during the evaluation process before generation.\nPass it directly to the :code:`preprocess_fn` key in your eval config dictionary.\n\nThe system injects into this function the batch data, as well as the RAG spec and \nthe prompt manager of an individual leaf config.\n\n\n.. py:function:: preprocess_fn(batch: dict[str, list], rag: RFLangChainRagSpec, prompt_manager: RFPromptManager) -> dict[str, list]\n\n   :param batch: Dictionary with a batch of examples with dataset field names as keys and lists as values\n   :type batch: dict[str, list]\n\n   :param rag: RAG specification object for document chunk retrieval and context serialization\n   :type rag: RFLangChainRagSpec\n\n   :param prompt_manager: Prompt manager object for handling instructions and few-shot examples\n   :type prompt_manager: RFPromptManager\n\n   :return: Dictionary with the preprocessed batch. It must have a reserved key :code:`"prompts"` for the fully formatted prompts for the generator. Other key-value pairs from the original batch can also be copied over if you want.\n   :rtype: dict[str, list]\n\n\n**Examples:**\n\n.. code-block:: python\n\n    # Example 1 from FiQA use case: RAG-based preprocessing with document chunk retrieval\n    # This example demonstrates how metadata fields ingested via metadata_func in the\n    # document loader (e.g., "corpus_id") are accessible on each Document object\'s\n    # .metadata dict after retrieval, enabling retrieval evaluation.\n    def sample_preprocess_fn(batch: dict[str, list], rag: RFLangChainRagSpec, prompt_manager: RFPromptManager) -> dict[str, list]:\n\t\t"""Function to prepare the final inputs given to the generator model"""\n \n\t\tINSTRUCTIONS = "Utilize your financial knowledge, give your answer or opinion to the input question or subject matter."\n \n\t\t# Perform batched retrieval over all queries; returns a list of lists of k documents per query\n\t\tall_context = rag.get_context(batch_queries=batch["query"], serialize=False)\n \n\t\t# Extract the retrieved document ids from the context.\n\t\t# The "corpus_id" metadata field was ingested via metadata_func in the document loader\n\t\t# (see RFLangChainRagSpec examples) and is now accessible on each Document object.\n\t\tretrieved_documents = [\n\t\t\t[doc.metadata["corpus_id"] for doc in docs] for docs in all_context\n\t\t]\n \n\t\t# Serialize the retrieved documents into a single string per query using the document_template.\n\t\t# If a custom document_template was provided in the RAG spec (e.g., to include title metadata),\n\t\t# it is applied here; otherwise the default "metadata:\\ncontent" template is used.\n\t\tserialized_context = rag.serialize_documents(all_context)\n\t\tbatch["query_id"] = [int(query_id) for query_id in batch["query_id"]]\n \n\t\t# Each batch to contain conversational prompt, retrieved documents, and original \'query_id\', \'query\', \'metadata\'\n\t\treturn {\n\t\t\t"prompts": [\n\t\t\t\t[\n\t\t\t\t\t{"role": "system", "content": INSTRUCTIONS},\n\t\t\t\t\t{\n\t\t\t\t\t\t"role": "user",\n\t\t\t\t\t\t"content": f"Here is some relevant context:\\n{context}. \\nNow answer the following question using the context provided earlier:\\n{question}",\n\t\t\t\t\t},\n\t\t\t\t]\n\t\t\t\tfor question, context in zip(batch["query"], serialized_context)\n\t\t\t],\n\t\t\t"retrieved_documents": retrieved_documents,\n\t\t\t**batch,\n\t\t}\n\n.. code-block:: python\n\n    # Example 2 from GSM8K use case: Few-shot learning preprocessing without RAG\n    def sample_preprocess_fn(batch: dict[str, list], rag: RFLangChainRagSpec, prompt_manager: RFPromptManager) -> dict[str, list]:\n\t\t"""Function to prepare the final inputs given to the generator model"""\n\n\t\treturn {\n\t\t\t"prompts": [\n\t\t\t\t[\n\t\t\t\t\t{"role": "system", "content": prompt_manager.get_instructions()},\n\t\t\t\t\t{\n\t\t\t\t\t\t"role": "user",\n\t\t\t\t\t\t"content": f"Here are some examples: \\n{examples}. \\nNow answer the following question:\\n{question}",\n\t\t\t\t\t},\n\t\t\t\t]\n\t\t\t\tfor question, examples in zip(\n\t\t\t\t\tbatch["question"],\n\t\t\t\t\tprompt_manager.get_fewshot_examples(user_queries=batch["question"]),\n\t\t\t\t)\n\t\t\t],\n\t\t\t**batch,\n\t\t}',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.1773, 0.4343],
#         [0.1773, 1.0000, 0.4398],
#         [0.4343, 0.4398, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 444 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 444 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 15 tokens mean: 41.97 tokens max: 70 tokens	min: 36 tokens mean: 225.67 tokens max: 256 tokens	min: 0.0 mean: 0.25 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  How does RapidFire AI's approach to multi-config experimentation unify training (run_fit) and evaluation (run_evals) workflows under a common adaptive execution model, and what are the key differences in how each workflow exposes parallelism controls, return values, and user-provided functions?	Step 5: Monitor training behaviors on ML metrics dashboard -------- .. raw:: html style="cursor: zoom-in; max-width: 100%;" onclick="this.requestFullscreen()"> Step 6: Interactive Control (IC) Ops: Stop, Clone-Modify; check their results ----- .. raw:: html style="cursor: zoom-in; max-width: 100%;" onclick="this.requestFullscreen()"> .. raw:: html style="cursor: zoom-in; max-width: 100%;" onclick="this.requestFullscreen()"> .. raw:: html style="cursor: zoom-in; max-width: 100%;" onclick="this.requestFullscreen()">	0.0
  How does the num_shards parameter in run_evals() relate to the online aggregation confidence interval computation, and what is the end-to-end flow from setting num_shards to seeing narrowing confidence intervals on eval metrics?	see its details on Hugging Face <https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized>__. We use a sample of 500 training examples for tractable demo runtimes.	0.0
  How does the num_shards parameter in run_evals() relate to the online aggregation confidence interval computation, and what is the end-to-end flow from setting num_shards to seeing narrowing confidence intervals on eval metrics?	RapidFire AI transforms the status quo by adapting the powerful idea of online aggregation from database systems research to LLM evals. Our adaptive execution engine, :doc:as described on this page</difference>, automatically shards the data and processes multiple configs in parallel, one shard at a time, with efficient swapping techniques. This means you get running metric estimates with confidence intervals in real time. So, you can confidently stop poor configs earlier, clone better configs on the fly, and perform more informed exploration to reach much better eval metrics in much less time. Example: Traditional Batch Evals vs. RapidFire AI ------- For instance, suppose you have an evals set with 400 queries. You decide to compare, say, 4 RAG configs in one go with RapidFire AI with number of shards set to 8. The illustration below contrasts traditional batch evals vs. RapidFire AI's approach for a simple eval metric. .. list-table::	1.0

• Loss: ContrastiveLoss with these parameters:

  {
      "distance_metric": "SiameseDistanceMetric.COSINE_DISTANCE",
      "margin": 0.5,
      "size_average": true
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 100
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 100
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
17.8571	500	0.0078
35.7143	1000	0.0032
53.5714	1500	0.0027
71.4286	2000	0.0024
89.2857	2500	0.0022

Training Time

• Training: 8.5 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ContrastiveLoss

@inproceedings{hadsell2006dimensionality,
    author={Hadsell, R. and Chopra, S. and LeCun, Y.},
    booktitle={2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06)},
    title={Dimensionality Reduction by Learning an Invariant Mapping},
    year={2006},
    volume={2},
    number={},
    pages={1735-1742},
    doi={10.1109/CVPR.2006.100}
}