kwondw/mpnet-base-all-nli-triplet

MPNet base trained on AllNLI triplets

This is a sentence-transformers model finetuned from microsoft/mpnet-base on the all-nli dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: microsoft/mpnet-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • all-nli
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'MPNetModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kwondw/mpnet-base-all-nli-triplet")
# Run inference
sentences = [
    'A construction worker peeking out of a manhole while his coworker sits on the sidewalk smiling.',
    'A worker is looking out of a manhole.',
    'The workers are both inside the manhole.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5718, 0.5598],
#         [0.5718, 1.0000, 0.5422],
#         [0.5598, 0.5422, 1.0000]])

Evaluation

Metrics

Triplet

• Dataset: all-nli-dev
• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	0.9154

Training Details

Training Dataset

all-nli

• Dataset: all-nli at d482672
• Size: 100,000 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 8 tokens mean: 14.59 tokens max: 30 tokens	min: 6 tokens mean: 13.96 tokens max: 28 tokens	min: 5 tokens mean: 15.1 tokens max: 30 tokens

• Samples:

  anchor	positive	negative
  A person on a horse jumps over a broken down airplane.	A person is outdoors, on a horse.	A person is at a diner, ordering an omelette.
  Children smiling and waving at camera	There are children present	The kids are frowning
  A boy is jumping on skateboard in the middle of a red bridge.	The boy does a skateboarding trick.	The boy skates down the sidewalk.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

all-nli

• Dataset: all-nli at d482672
• Size: 6,584 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 7 tokens mean: 17.57 tokens max: 36 tokens	min: 5 tokens mean: 9.98 tokens max: 28 tokens	min: 5 tokens mean: 10.38 tokens max: 18 tokens

• Samples:

  anchor	positive	negative
  Two women are embracing while holding to go packages.	Two woman are holding packages.	The men are fighting outside a deli.
  Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink.	Two kids in numbered jerseys wash their hands.	Two kids in jackets walk to school.
  A man selling donuts to a customer during a world exhibition event held in the city of Angeles	A man selling donuts to a customer.	A woman drinks her coffee in a small cafe.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.1
• fp16: True
• per_device_eval_batch_size: 16
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• per_device_train_batch_size: 16
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	all-nli-dev_cosine_accuracy
-1	-1	-	-	0.6211
0.016	100	3.1300	1.4117	0.7506
0.032	200	1.5028	0.7034	0.8044
0.048	300	1.1036	0.6180	0.8247
0.064	400	1.0011	0.6082	0.8253
0.08	500	0.9326	0.5667	0.8372
0.096	600	0.9038	0.5407	0.8501
0.112	700	0.8183	0.5173	0.8548
0.128	800	0.8704	0.5138	0.8638
0.144	900	0.8125	0.4933	0.8688
0.16	1000	0.7368	0.4607	0.8770
0.176	1100	0.6957	0.4930	0.8806
0.192	1200	0.7251	0.4160	0.8972
0.208	1300	0.6322	0.4211	0.8931
0.224	1400	0.6312	0.4382	0.8893
0.24	1500	0.6207	0.4303	0.8872
0.256	1600	0.5900	0.4217	0.8903
0.272	1700	0.5724	0.4101	0.8978
0.288	1800	0.6214	0.3930	0.9049
0.304	1900	0.6223	0.3848	0.9007
0.32	2000	0.5789	0.3959	0.8998
0.336	2100	0.5334	0.3794	0.9033
0.352	2200	0.4986	0.3804	0.8988
0.368	2300	0.5138	0.3787	0.9026
0.384	2400	0.5372	0.3827	0.9048
0.4	2500	0.5093	0.3773	0.9029
0.416	2600	0.4898	0.3765	0.9034
0.432	2700	0.5248	0.3842	0.9040
0.448	2800	0.5014	0.3754	0.9067
0.464	2900	0.4801	0.3785	0.9034
0.48	3000	0.4678	0.3668	0.9074
0.496	3100	0.4869	0.3606	0.9067
0.512	3200	0.4709	0.3712	0.9069
0.528	3300	0.4705	0.3737	0.9036
0.544	3400	0.4548	0.3696	0.9014
0.56	3500	0.4678	0.3558	0.9075
0.576	3600	0.4083	0.3563	0.9104
0.592	3700	0.4366	0.3588	0.9086
0.608	3800	0.4224	0.3540	0.9101
0.624	3900	0.4056	0.3549	0.9108
0.64	4000	0.3687	0.3543	0.9119
0.656	4100	0.3982	0.3518	0.9078
0.672	4200	0.4383	0.3477	0.9099
0.688	4300	0.4362	0.3443	0.9140
0.704	4400	0.3842	0.3491	0.9134
0.72	4500	0.4117	0.3441	0.9098
0.736	4600	0.3886	0.3440	0.9128
0.752	4700	0.3776	0.3450	0.9157
0.768	4800	0.4158	0.3397	0.9119
0.784	4900	0.3924	0.3408	0.9125
0.8	5000	0.3763	0.3425	0.9121
0.816	5100	0.3689	0.3428	0.9134
0.832	5200	0.4122	0.3415	0.9130
0.848	5300	0.3442	0.3377	0.9142
0.864	5400	0.3567	0.3342	0.9142
0.88	5500	0.3597	0.3383	0.9136
0.896	5600	0.3571	0.3351	0.9145
0.912	5700	0.3519	0.3332	0.9143
0.928	5800	0.3305	0.3323	0.9145
0.944	5900	0.4017	0.3317	0.9145
0.96	6000	0.3943	0.3322	0.9140
0.976	6100	0.3530	0.3320	0.9149
0.992	6200	0.1762	0.3319	0.9154

Training Time

• Training: 1.9 hours

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.0
• Transformers: 5.13.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}