Stergios-Konstantinidis/MNLP_M3_tokenizer_tuned_pos_neg_3

SentenceTransformer based on sucharush/e5_stem_finetuned

This is a sentence-transformers model finetuned from sucharush/e5_stem_finetuned. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sucharush/e5_stem_finetuned
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Stergios-Konstantinidis/MNLP_M3_tokenizer_tuned_pos_neg_3")
# Run inference
sentences = [
    "Question: who played granny on the beverly hillbillies tv show, Answer: Irene Ryan Ryan is most widely known for her portrayal of Granny, the mother-in-law of Buddy Ebsen's character, on the long-running TV series The Beverly Hillbillies (1962–1971), for which she was nominated for Emmy Awards for Outstanding Lead Actress in a Comedy Series in 1963 and 1964.",
    '[{\'from\': \'human\', \'value\': \'Solve the following math problem step-by-step.\\nSimplify your answer as much as possible. Present your final answer as \\\\boxed{Your Answer}.\\nHenley bought 300 candies and shared them with her two brothers. However, they realized 40% of them were sour, and they had to separate those out. If they then shared the good candies equally, how many candies did each get?\'}, {\'from\': \'gpt\', \'value\': "Step 1: First, let\'s calculate the number of sour candies. sour_candies = 300 * 0.4 = 120\\nStep 2: To find the number of good candies, we subtract the number of sour candies from the total number of candies. good_candies = 300 - 120 = 180\\nStep 3: Since Henley shared the candies equally with her two brothers, we need to divide the number of good candies by 3 (Henley and her two brothers). candies_per_person = good_candies / 3 = 180 / 3 = 60\\n\\nAnswer:\\n\\\\boxed{60}"}]',
    '[{\'from\': \'human\', \'value\': \'Solve the following math problem step-by-step.\\nSimplify your answer as much as possible. Present your final answer as \\\\boxed{Your Answer}.\\nThe bowl of fruit contains apples, pears, and bananas. There are two more pears than apples, and three more bananas than pears.  If the bowl contains 19 pieces of fruit, how many bananas does it contain?\'}, {\'from\': \'gpt\', \'value\': "Step 1: Let\'s denote the number of apples as A, the number of pears as P, and the number of bananas as B.\\nStep 2: According to the given information, there are two more pears than apples, so P = A + 2.\\nStep 3: Also, there are three more bananas than pears, so B = P + 3.\\nStep 4: The total number of fruits in the bowl is given as 19, so A + P + B = 19.\\nStep 5: Substituting the values of P and B from steps 2 and 3 into the equation in step 4, we get A + (A + 2) + ((A + 2) + 3) = 19.\\nStep 6: Simplifying the equation, 3A + 7 = 19.\\nStep 7: Solving for A, 3A = 12 => A = 4.\\nStep 8: Substituting the value of A back into the equations in steps 2 and 3, we get P = 4 + 2 = 6 and B = 6 + 3 = 9.\\nStep 9: Therefore, the bowl contains 9 bananas.\\n\\nAnswer:\\n\\\\boxed{9}"}]',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 60,000 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	int
  details	min: 16 tokens mean: 177.36 tokens max: 510 tokens	min: 16 tokens mean: 179.6 tokens max: 512 tokens	0: ~66.60% 1: ~33.40%

• Samples:

  sentence_0	sentence_1	label
  Question: where is nigeria located in the map of africa, Answer: Nigeria The Federal Republic of Nigeria /naɪˈdʒɪəriə/ ( listen), commonly referred to as Nigeria, is a federal republic in West Africa, bordering Benin in the west, Chad and Cameroon in the east, and Niger in the north. Its coast in the south lies on the Gulf of Guinea in the Atlantic Ocean. It comprises 36 states and the Federal Capital Territory, where the capital, Abuja is located. Nigeria is officially a democratic secular country.[6]	Question: where is nigeria located in the map of africa, Answer: Nigeria The Federal Republic of Nigeria /naɪˈdʒɪəriə/ ( listen), commonly referred to as Nigeria, is a federal republic in West Africa, bordering Benin in the west, Chad and Cameroon in the east, and Niger in the north. Its coast in the south lies on the Gulf of Guinea in the Atlantic Ocean. It comprises 36 states and the Federal Capital Territory, where the capital, Abuja is located. Nigeria is officially a democratic secular country.[6]	1
  Question: who was the last person to live in biltmore house, Answer: Biltmore Estate After the divorce of the Cecils in 1934, Cornelia left the estate never to return; however, John Cecil maintained his residence in the Bachelors' Wing until his death in 1954. Their eldest son George Henry Vanderbilt Cecil, occupied rooms in the wing until 1956. At this point Biltmore House ceased to be a family residence and has continued to be operated as a historic house museum.	[{'from': 'human', 'value': 'What is the charge associated with the electron cloud, and how does it relate to the overall charge of an atom?'}, {'from': 'gpt', 'value': "The electron cloud in an atom carries a negative charge. This charge is balanced by the positive charge of the atomic nucleus, which results in a neutral atom when both charges are equal. If there is an excess or deficiency of electrons, the atom becomes an ion and carries a net positive or negative charge.\n\nIt is worth noting that the historical convention of assigning electrons a negative charge can sometimes lead to confusion. If we were to start anew, it might be logical to designate electrons as positively charged and the nucleus as negatively charged. However, it's important to remember this convention when performing electronic calculations to avoid errors due to incorrect charge assignments."}]	0
  [{'from': 'human', 'value': 'Solve the following math problem step-by-step.\nSimplify your answer as much as possible. Present your final answer as \boxed{Your Answer}.\nJames buys jars for his honey. He has 5 hives that each produce 20 liters of honey. Each jar can hold .5 liters. How many jars will he need to buy if his friend is bringing his own jars for half the honey?'}, {'from': 'gpt', 'value': "Step 1: First, we need to determine the total amount of honey produced by James' 5 hives. Each hive produces 20 liters of honey, so the total honey produced is 5 * 20 = 100 liters.\nStep 2: Since his friend is bringing his own jars for half of the honey, James will only keep half of the total honey produced, which is 100 / 2 = 50 liters.\nStep 3: Now, we need to find out how many jars James needs to hold 50 liters of honey, given that each jar can hold 0.5 liters.\nStep 4: To calculate the number of jars needed, we can simply divide the total amount of honey (50 liters) by the cap...	Question: what does ds stand for in uk police, Answer: Criminal investigation department In the United Kingdom, smaller police stations usually have more uniformed officers than CID officers, typically five DCs with a detective sergeant (DS) in overall command. In larger stations many DCs, DSs and detective inspectors are present under the overall responsibility of a detective chief inspector.	0

• Loss: ContrastiveTensionLoss

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 15
• per_device_eval_batch_size: 15
• num_train_epochs: 1
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 15
• per_device_eval_batch_size: 15
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss
0.125	500	13.9952
0.25	1000	11.0821
0.375	1500	9.5046
0.5	2000	9.6757
0.625	2500	9.4813
0.75	3000	9.522
0.875	3500	9.6432
1.0	4000	9.6282

Framework Versions

• Python: 3.12.8
• Sentence Transformers: 3.4.1
• Transformers: 4.52.4
• PyTorch: 2.6.0+cu126
• Accelerate: 1.3.0
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

ContrastiveTensionLoss

@inproceedings{carlsson2021semantic,
    title={Semantic Re-tuning with Contrastive Tension},
    author={Fredrik Carlsson and Amaru Cuba Gyllensten and Evangelia Gogoulou and Erik Ylip{"a}{"a} Hellqvist and Magnus Sahlgren},
    booktitle={International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=Ov_sMNau-PF}
}