TesNik369/bge-m3-tqa-lesson-split-merged

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3 on the tqa-bge-m3-triplets-lesson-split dataset. It maps inputs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • tqa-bge-m3-triplets-lesson-split

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("TesNik369/bge-m3-tqa-lesson-split-merged")
# Run inference
queries = [
    'differentiation',
]
documents = [
    'When Earth was entirely molten, gravity drew denser elements to the center and lighter elements rose to the surface. The separation of Earth into layers based on density is known as differentiation. The densest material moved to the center to create the planets dense metallic core. Materials that are intermediate in density became part of the mantle (Figure 1.1).',
    '1. Distinctive rock formations may be recognizable across large regions (Figure 1.1).',
    'The mesosphere is the layer above the stratosphere. It rises to about 85 kilometers (53 miles) above the surface. Temperature decreases with altitude in this layer.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.5101, 0.5188, 0.3545]])

Evaluation

Metrics

Information Retrieval

• Dataset: tqa-dev-lesson-disjoint
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.5303
cosine_accuracy@3	0.7778
cosine_accuracy@5	0.8485
cosine_accuracy@10	0.899
cosine_precision@1	0.5303
cosine_precision@3	0.346
cosine_precision@5	0.2808
cosine_precision@10	0.1992
cosine_recall@1	0.1243
cosine_recall@3	0.2431
cosine_recall@5	0.3144
cosine_recall@10	0.4124
cosine_ndcg@10	0.3994
cosine_mrr@10	0.6666
cosine_map@100	0.3157

Training Details

Training Dataset

tqa-bge-m3-triplets-lesson-split

• Dataset: tqa-bge-m3-triplets-lesson-split
• Size: 8,257 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 3 tokens mean: 15.04 tokens max: 34 tokens	min: 23 tokens mean: 212.56 tokens max: 512 tokens	min: 34 tokens mean: 203.17 tokens max: 512 tokens

• Samples:

  anchor	positive	negative
  covalent bonds are weak; it does not take a lot of energy to break them apart.	Ions come together to create a molecule so that electrical charges are balanced; the positive charges balance the negative charges and the molecule has no electrical charge. To balance electrical charge, an atom may share its electron with another atom, give it away, or receive an electron from another atom. The joining of ions to make molecules is called chemical bonding. There are three main types of chemical bonds that are important in our discussion of minerals and rocks: Ionic bond: Electrons are transferred between atoms. An ion will give one or more electrons to another ion. Table salt, sodium chloride (NaCl), is a common example of an ionic compound. Note that sodium is on the left side of the periodic table and that chlorine is on the right side of the periodic table. In the Figure 1.2, an atom of lithium donates an electron to an atom of fluorine to form an ionic compound. The transfer of the electron gives the lithium ion a net charge of +1, and the fluorine ion a net charge...	A covalent bond is the force of attraction that holds together two atoms that share a pair of valence electrons. The shared electrons are attracted to the nuclei of both atoms. This forms a molecule consisting of two or more atoms. Covalent bonds form only between atoms of nonmetals.
  The human species first evolved about	The earliest members of the human species evolved around 200,000 years ago in Africa. Early humans lived in small populations of nomadic hunters and gatherers. Human beings remained in Africa until about 40,000 years ago. After that, they spread throughout Europe, Asia, and Australia. By 10,000 years ago, the first human beings colonized the Americas. During this long period of time, the total number of human beings increased very slowly. Birth rates were fairly high but so were death rates, producing low rates of population growth. Human beings invented agriculture about 10,000 years ago. This provided a bigger, more dependable food supply. It also allowed people to settle down in villages and cities for the first time. Birth rates went up because there was more food and settled life had other advantages. Death rates also rose because of crowded living conditions and diseases that spread from domestic animals. Because the higher birth rates were matched by higher death rates, the huma...	Humans evolved from primates, and apes and humans have a primate common ancestor. About 7 million years ago, chimpanzees (our closest living relatives) and humans shared their last common ancestor.
  hormones travel around your body in your blood.	Your cardiovascular system has many jobs. At times the cardiovascular system can work like a pump, a heating system, or even a postal carrier. To do these tasks, your cardiovascular system works with other organ systems, such as the respiratory, endocrine, and nervous systems. The cardiovascular system (Figure 1.1) is made up of the heart, the blood vessels, and the blood. It moves nutrients, gases (like oxygen), and wastes to and from your cells. Every cell in your body depends on your cardiovascular system. If your cells dont receive nutrients, they cannot survive. The main function of the cardiovascular system is to deliver oxygen to each of your cells. Blood receives oxygen in your lungs (the main organs of the respiratory system) and then is pumped, by your heart, throughout your body. The oxygen then diffuses into your cells, and carbon dioxide, a waste product of cellular respiration, moves from your cells into your blood to be delivered back to your lungs and exhaled. Each cell...	Endocrine hormones travel throughout the body in the blood. However, each endocrine hormone affects only certain cells, called target cells.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 2
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• warmup_steps: 0.05
• gradient_accumulation_steps: 8
• fp16: True
• per_device_eval_batch_size: 32
• seed: 3407
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 2
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: None
• warmup_steps: 0.05
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 8
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 3407
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

Training Logs

Epoch	Step	Training Loss	tqa-dev-lesson-disjoint_cosine_ndcg@10
-1	-1	-	0.3891
0.0155	1	1.1003	-
0.3095	20	1.0544	-
0.6190	40	1.0563	-
0.9284	60	1.0275	-
1.0	65	-	0.3970
1.2321	80	1.0126	-
1.5416	100	0.9894	-
1.8511	120	0.9915	-
2.0	130	-	0.3994
-1	-1	-	0.3994

Training Time

• Training: 1.1 hours

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 6.1.0
• Transformers: 5.19.0
• PyTorch: 2.8.0+cu126
• Accelerate: 1.11.0
• Datasets: 5.1.0
• Tokenizers: 0.23.3

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}