Shailu1492/roberta-base-mnrl-snli

SentenceTransformer based on FacebookAI/roberta-base

This is a sentence-transformers model finetuned from FacebookAI/roberta-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: FacebookAI/roberta-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RobertaModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Shailu1492/roberta-base-mnrl-snli")
# Run inference
sentences = [
    "It's very cold at the top of the mountains, where the peaks are covered by clouds.",
    "It's very cold at the top of the mountains, where the peaks are covered by clouds.",
    'The man is organizing.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  1.0000, -0.0760],
#         [ 1.0000,  1.0000, -0.0760],
#         [-0.0760, -0.0760,  1.0000]])

Evaluation

Metrics

Semantic Similarity

• Dataset: stsb_valid
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.7302
spearman_cosine	0.7308

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,148,085 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 6 tokens mean: 14.89 tokens max: 59 tokens	min: 6 tokens mean: 14.89 tokens max: 59 tokens

• Samples:

  anchor	positive
  They wanted to make the neighborhood feel like a community.	They wanted to make the neighborhood feel like a community.
  The procedures will block foreign corporations from offering satellite coverage.	The procedures will block foreign corporations from offering satellite coverage.
  Wayne starred in over 50 films.	Wayne starred in over 50 films.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• num_train_epochs: 4
• learning_rate: 2e-05
• bf16: True
• dataloader_drop_last: True

All Hyperparameters

Click to expand

• per_device_train_batch_size: 128
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	stsb_valid_spearman_cosine
0.0557	500	0.0375	0.8074
0.1115	1000	0.0001	0.7911
0.1672	1500	0.0001	0.7826
0.2230	2000	0.0001	0.7695
0.2787	2500	0.0001	0.8121
0.3345	3000	0.0001	0.7634
0.3902	3500	0.0000	0.7860
0.4460	4000	0.0000	0.7601
0.5017	4500	0.0000	0.7915
0.5575	5000	0.0001	0.7655
0.6132	5500	0.0001	0.7542
0.6690	6000	0.0000	0.7636
0.7247	6500	0.0000	0.7467
0.7805	7000	0.0001	0.7563
0.8362	7500	0.0001	0.7610
0.8920	8000	0.0000	0.7314
0.9477	8500	0.0001	0.7707
1.0035	9000	0.0000	0.7652
1.0592	9500	0.0000	0.7541
1.1150	10000	0.0000	0.7655
1.1707	10500	0.0001	0.7720
1.2264	11000	0.0000	0.7621
1.2822	11500	0.0000	0.7576
1.3379	12000	0.0000	0.7665
1.3937	12500	0.0000	0.7597
1.4494	13000	0.0000	0.7707
1.5052	13500	0.0000	0.7537
1.5609	14000	0.0000	0.7520
1.6167	14500	0.0000	0.7574
1.6724	15000	0.0000	0.7627
1.7282	15500	0.0000	0.7549
1.7839	16000	0.0000	0.7633
1.8397	16500	0.0000	0.7574
1.8954	17000	0.0000	0.7517
1.9512	17500	0.0000	0.7307
2.0069	18000	0.0000	0.7356
2.0627	18500	0.0000	0.7378
2.1184	19000	0.0000	0.7537
2.1742	19500	0.0000	0.7442
2.2299	20000	0.0000	0.7421
2.2857	20500	0.0000	0.7517
2.3414	21000	0.0000	0.7494
2.3971	21500	0.0000	0.7513
2.4529	22000	0.0000	0.7463
2.5086	22500	0.0000	0.7458
2.5644	23000	0.0000	0.7259
2.6201	23500	0.0000	0.7337
2.6759	24000	0.0000	0.7330
2.7316	24500	0.0000	0.7306
2.7874	25000	0.0000	0.7317
2.8431	25500	0.0000	0.7285
2.8989	26000	0.0000	0.7301
2.9546	26500	0.0000	0.7220
3.0104	27000	0.0000	0.7302
3.0661	27500	0.0000	0.7239
3.1219	28000	0.0000	0.7233
3.1776	28500	0.0000	0.7360
3.2334	29000	0.0000	0.7152
3.2891	29500	0.0000	0.7298
3.3449	30000	0.0000	0.7323
3.4006	30500	0.0000	0.7183
3.4563	31000	0.0000	0.7277
3.5121	31500	0.0000	0.7300
3.5678	32000	0.0000	0.7332
3.6236	32500	0.0000	0.7384
3.6793	33000	0.0000	0.7375
3.7351	33500	0.0000	0.7363
3.7908	34000	0.0000	0.7358
3.8466	34500	0.0000	0.7333
3.9023	35000	0.0000	0.7314
3.9581	35500	0.0000	0.7309
4.0	35876	-	0.7308

Training Time

• Training: 7.8 hours
• Evaluation: 4.0 minutes
• Total: 7.9 hours

Framework Versions

• Python: 3.10.20
• Sentence Transformers: 5.6.0
• Transformers: 5.14.1
• PyTorch: 2.13.0+cu130
• Accelerate: 1.10.1
• Datasets: 4.8.4
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}