seregadgl/splade_gemma_google_base_checkpoint_100_ver2

SentenceTransformer based on seregadgl/splade_gemma_google_base_checkpoint_100_clear

This is a sentence-transformers model finetuned from seregadgl/splade_gemma_google_base_checkpoint_100_clear on the car_and_product_triplet_103k dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: seregadgl/splade_gemma_google_base_checkpoint_100_clear
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • car_and_product_triplet_103k

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): SparseLayer(
    (linear): Linear(in_features=768, out_features=262144, bias=True)
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("seregadgl/splade_gemma_google_base_checkpoint_100_ver2")
# Run inference
sentences = [
    'query: 1452634 santool jawa 300 cl',
    'document: 1452634 santool съемник для сальников jawa 300 cl',
    'document: 1453934 santool съемник для сальников jawa 300 cl',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.1443, 0.1452],
#         [0.1443, 1.0000, 0.7490],
#         [0.1452, 0.7490, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: val_set_fine
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.742
cosine_precision@1	0.742
cosine_precision@3	0.2763
cosine_precision@5	0.1728
cosine_precision@10	0.0891
cosine_recall@1	0.742
cosine_recall@3	0.829
cosine_recall@5	0.864
cosine_recall@10	0.891
cosine_ndcg@10	0.8161
cosine_mrr@10	0.7919
cosine_map@100	0.7956

Training Details

Training Dataset

car_and_product_triplet_103k

• Dataset: car_and_product_triplet_103k at 3519181
• Size: 102,127 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 5 tokens mean: 16.27 tokens max: 44 tokens	min: 4 tokens mean: 23.62 tokens max: 77 tokens	min: 6 tokens mean: 23.2 tokens max: 47 tokens

• Samples:

  anchor	positive	negative
  query: погружной блендер tefal optichef hb64f810	document: погружной блендер тефаль optichef hb64f810	document: погружной миксер tefal mixchef hb64f850
  query: 375675836 niteo	document: тосол 375675836 для ford f350 полуночный синий	document: тосол 375625836 для ford f350 полуночный синий фиалковый
  query: накидка с подогревом dodge viper pink	document: накидка с подогревом acdelco арт 787327sx dodge viper розовый	document: 787327sx накидка с подогревом indian challenger лаймовый

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
      "document_regularizer_weight": 1e-05,
      "query_regularizer_weight": 1e-05
  }
  

Evaluation Dataset

car_and_product_triplet_103k

• Dataset: car_and_product_triplet_103k at 3519181
• Size: 1,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 5 tokens mean: 16.73 tokens max: 74 tokens	min: 5 tokens mean: 23.54 tokens max: 80 tokens	min: 5 tokens mean: 22.66 tokens max: 65 tokens

• Samples:

  anchor	positive	negative
  query: зеркала для 'слепых' зон volkswagen arteon	document: зеркала для 'слепых' зон 86635985zz для volkswagen arteon перламутровочёрный	document: 86635985zz зеркала для 'слепых' зон иж юпитер2 голубой
  query: elf bar lux 1500 лимонад голубой малины 1500	document: одноразовая электронная сигарета эльф бар 1 5000 мл lemonade blue raspberry 340440526	document: elf bar vibe 1000 мохито зелёного яблока 1000
  query: удалитель наклеек chevrolet corvette onyx	document: удалитель наклеек 20810588pl для chevrolet corvette оникс	document: удалитель наклеек 20810588pl для maserati levante янтарный

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
      "document_regularizer_weight": 1e-05,
      "query_regularizer_weight": 1e-05
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• gradient_accumulation_steps: 16
• learning_rate: 0.0001
• num_train_epochs: 1
• warmup_steps: 10
• fp16: True
• load_best_model_at_end: True
• router_mapping: {'query': 'anchor', 'document': 'positive'}

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 16
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.0001
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 10
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {'query': 'anchor', 'document': 'positive'}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Validation Loss	val_set_fine_cosine_ndcg@10
0.0125	10	0.8461	0.7841
0.0251	20	0.8195	0.8009
0.0376	30	0.7884	0.7967
0.0501	40	0.7641	0.8097
0.0627	50	0.7503	0.8146
0.0752	60	0.7140	0.8151
0.0877	70	0.7165	0.8180
0.1003	80	0.6955	0.8131
0.1128	90	0.6866	0.8157
0.1253	100	0.6735	0.8170
0.1379	110	0.6766	0.8159
0.1504	120	0.6609	0.8161

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.2
• Transformers: 4.57.1
• PyTorch: 2.8.0+cu126
• Accelerate: 1.11.0
• Datasets: 4.4.2
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}