albertus-sussex/veriscrape-sbert-auto-reference_5_to_verify_5-fold-1

SentenceTransformer based on Alibaba-NLP/gte-base-en-v1.5

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Alibaba-NLP/gte-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: NewModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("albertus-sussex/veriscrape-sbert-auto-reference_5_to_verify_5-fold-1")
# Run inference
sentences = [
    '2010 Subaru Legacy Overview',
    '2011 Honda CR-Z Overview',
    '10 mpg',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	1.0

Silhouette

• Evaluated with veriscrape.training.SilhouetteEvaluator

Metric	Value
silhouette_cosine	0.9427
silhouette_euclidean	0.7829

Triplet

• Evaluated with TripletEvaluator

Metric	Value
cosine_accuracy	1.0

Silhouette

• Evaluated with veriscrape.training.SilhouetteEvaluator

Metric	Value
silhouette_cosine	0.9422
silhouette_euclidean	0.7823

Training Details

Training Dataset

Unnamed Dataset

• Size: 8,609 training samples
• Columns: anchor, positive, negative, pos_attr_name, and neg_attr_name
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative	pos_attr_name	neg_attr_name
  type	string	string	string	string	string
  details	min: 3 tokens mean: 11.36 tokens max: 35 tokens	min: 3 tokens mean: 11.35 tokens max: 35 tokens	min: 4 tokens mean: 11.27 tokens max: 35 tokens	min: 3 tokens mean: 3.59 tokens max: 5 tokens	min: 3 tokens mean: 3.45 tokens max: 5 tokens

• Samples:

  anchor	positive	negative	pos_attr_name	neg_attr_name
  15 mpg	21 mpg City, 31 mpg Hwy	2011 Toyota Prius	fuel_economy	model
  3.7L I5, 20 valves, 242 hp @ 5600 rpm	175-hp, 2.4-liter I-4 (regular gas)	2011 Lexus LS 460	engine	model
  City: 12 – 18 Highway: 19 – 26	14 - 16	2010 Chevrolet Suburban	fuel_economy	model

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
      "triplet_margin": 5
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 957 evaluation samples
• Columns: anchor, positive, negative, pos_attr_name, and neg_attr_name
• Approximate statistics based on the first 957 samples:

  	anchor	positive	negative	pos_attr_name	neg_attr_name
  type	string	string	string	string	string
  details	min: 3 tokens mean: 11.41 tokens max: 35 tokens	min: 3 tokens mean: 11.19 tokens max: 35 tokens	min: 4 tokens mean: 10.87 tokens max: 35 tokens	min: 3 tokens mean: 3.54 tokens max: 5 tokens	min: 3 tokens mean: 3.53 tokens max: 5 tokens

• Samples:

  anchor	positive	negative	pos_attr_name	neg_attr_name
  2.4L I4, 16 valves, 176 hp @ 6000 rpm	3.0L I6, 24 valves, 230 hp @ 6500 rpm	MSRP: $17,520	engine	price
  433-hp, 4.7-liter V-8 (premium)	3.5L Gas V6, 285 HP	16 mpg	engine	fuel_economy
  MSRP: $48,040	$13,300 - $17,195	Engine: 2.0L Duratec in-linefour-cylinder DOHC and four valves per cylinder	price	engine

• Loss: TripletLoss with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
      "triplet_margin": 5
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• num_train_epochs: 5
• warmup_ratio: 0.1

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	cosine_accuracy	silhouette_cosine
-1	-1	-	-	0.7670	0.3073
1.0	68	0.2917	0.0	1.0	0.9426
2.0	136	0.0	0.0	1.0	0.9427
3.0	204	0.0	0.0	1.0	0.9427
4.0	272	0.0	0.0	1.0	0.9427
5.0	340	0.0	0.0	1.0	0.9427
-1	-1	-	-	1.0	0.9422

Framework Versions

• Python: 3.10.16
• Sentence Transformers: 4.0.1
• Transformers: 4.45.2
• PyTorch: 2.5.1+cu124
• Accelerate: 1.5.2
• Datasets: 3.1.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}