Hugging Face's logo

rdxtremity
/
search-stack-v1

Joblib
ONNX
Safetensors
English
Arabic
retrieval
search
lightgbm
cross-encoder
bge-m3
Model card Files
xet
 Community
search-stack-v1 / splade_v6_v2 /README.md
rdxtremity's picture
rdxtremity
add SPLADE v6_v2 (qwen30-pairs A/B winner, sparse nDCG 0.8189)
b737adb verified
preview code
|
Raw
History Blame Contribute Delete
21.7 kB
metadata 
tags:
  - sentence-transformers
  - sparse-encoder
  - sparse
  - splade
  - generated_from_trainer
  - dataset_size:1298215
  - loss:SpladeLoss
  - loss:SparseMultipleNegativesRankingLoss
  - loss:FlopsLoss
base_model: distilbert/distilbert-base-multilingual-cased
widget:
  - text: >-
      [Clan London] Women's Logo Tote Bag Olive (Olive) | حقيبة توت بشعار
      العلامة التجارية للنساء، زيتي (زيتي). Category: Bags > Shoppers & Totes.
  - text: >-
      [Fire & Glory] Women's Animal Print Midi Dress Light Gray (Light Gray) |
      فستان ميدي بأكمام طويلة وطبعة جرافيك للنساء، أسود (رمادي فاتح). Category:
      Dresses > Midi Dresses.
  - text: >-
      [melissa] Melissa Becky AD Pink (Pink) | ميليسا بيكي إيه دي (وردي).
      Category: Shoes > Slides & Flip-Flops.
  - text: >-
      [H&M] Curvy Ultra High-Rise Wide-Leg Jeans (Pastel Blue) | جينز كيرفي بخصر
      مرتفع جداً وأرجل واسعة (أزرق باستيل). Category: Denim > Jeans.
  - text: >-
      [Splash Basics] Flexi Comfort Drawstring A-line Midi Skirt Blue (Blue) |
      الرباط ألف خط تنورة ميدي (أزرق). Category: Bottoms > Skirts.
pipeline_tag: feature-extraction
library_name: sentence-transformers

SPLADE Sparse Encoder

This is a SPLADE Sparse Encoder model finetuned from distilbert/distilbert-base-multilingual-cased on the pairs and triplets datasets using the sentence-transformers library. It maps sentences & paragraphs to a 119547-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

  • Model Type: SPLADE Sparse Encoder
  • Base model: distilbert/distilbert-base-multilingual-cased
  • Maximum Sequence Length: 128 tokens
  • Output Dimensionality: 119547 dimensions
  • Similarity Function: Dot Product
  • Supported Modality: Text
  • Training Datasets:
    • pairs
    • triplets

Model Sources

Full Model Architecture 

SparseEncoder(
  (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 119547})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("sparse_encoder_model_id")
# Run inference
sentences = [
    'شنطة زهّانة',
    '[PANDORA] Aquarius Zodiac Charm (Silver) | دلاية برج الدلو (فضي). Category:  > Accessories.',
    "[Hurley] Women's Crew Neck Solid Sweatshirt Dark Salmon (Dark Salmon) | سويت شيرت نسائي سادة بياقة دائرية، برتقالي (سلموني غامق). Category: Hoodies & Sweatshirt > Sweatshirts.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 119547]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 11.9930,  13.0235,   7.8521],
#         [ 13.0235, 491.0651,  65.4858],
#         [  7.8521,  65.4858, 246.8788]])

Training Details

Training Datasets

pairs

  • Dataset: pairs
  • Size: 1,292,307 training samples
  • Columns: query and positive
  • Approximate statistics based on the first 100 samples:
    query positive
    type string string
    modality text text
    details
    • min: 6 tokens
    • mean: 10.33 tokens
    • max: 20 tokens
    • min: 28 tokens
    • mean: 51.14 tokens
    • max: 124 tokens
  • Samples:
    query positive
    men's watch analog [CASIO] Leather Strap Analog Watch (Mint Green) | ساعة انالوج جلد اصلي (أخضر نعناعي). Category: > .
    black hoodie women [Tribe of 6] Women's Logo Hooded Sweatshirt Black (Black) | سويت شيرت نسائي سادة بأكمام طويلة وغطاء رأس، أسود (أسود). Category: Hoodies & Sweatshirt > Hoodies.
    وشاح مخطط للنساء [MANGO] Geometric Stripe Scarf Beige (Beige) | وشاح بنقوش هندسية مخططة (بيج). Category: Accessories > Scarves.
  • Loss: SpladeLoss with these parameters:
    {
    "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
    "document_regularizer_weight": 3e-05,
    "query_regularizer_weight": 5e-05
}

triplets

  • Dataset: triplets
  • Size: 5,908 training samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 100 samples:
    query positive negative
    type string string string
    modality text text text
    details
    • min: 5 tokens
    • mean: 8.75 tokens
    • max: 17 tokens
    • min: 39 tokens
    • mean: 57.33 tokens
    • max: 82 tokens
    • min: 38 tokens
    • mean: 54.78 tokens
    • max: 88 tokens
  • Samples:
    query positive negative
    don't call me jennyfer jumpsuits & bodysuits [Don't Call Me Jennyfer] Women's One Sleeve Bodysuit Green (Green) | بودي سوت سادة بأكمام قصيرة للنساء، أخضر (أخضر). Category: Jumpsuits & Bodysuits > Body Suits. [Ashita Fernandes] Kenza Cupro V-Neck Jumpsuit Royal Blue (Facebook Blue) | كينزا جمبسوت كوبرو بفتحة رقبة V أزرق ملكي (أزرق فيسبوك). Category: Jumpsuits & Bodysuits > Jumpsuits.
    don't call me jennyfer jumpsuits & bodysuits [Don't Call Me Jennyfer] Women's One Sleeve Bodysuit Green (Green) | بودي سوت سادة بأكمام قصيرة للنساء، أخضر (أخضر). Category: Jumpsuits & Bodysuits > Body Suits. [Splash Intimates] Mesh Panel Shaping Bodysuit (Black) | بودي سوت مشد بأجزاء شبكية (أسود). Category: Jumpsuits & Bodysuits > Body Suits.
    don't call me jennyfer jumpsuits & bodysuits [Don't Call Me Jennyfer] Women's One Sleeve Bodysuit Green (Green) | بودي سوت سادة بأكمام قصيرة للنساء، أخضر (أخضر). Category: Jumpsuits & Bodysuits > Body Suits. [Ashita Fernandes] Kenza Cupro V-Neck Jumpsuit Royal Blue (Facebook Blue) | كينزا جمبسوت كوبرو بفتحة رقبة V أزرق ملكي (أزرق فيسبوك). Category: Jumpsuits & Bodysuits > Jumpsuits.
  • Loss: SpladeLoss with these parameters:
    {
    "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
    "document_regularizer_weight": 3e-05,
    "query_regularizer_weight": 5e-05
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 16
  • num_train_epochs: 1
  • learning_rate: 2e-05
  • warmup_steps: 0.1
  • gradient_accumulation_steps: 4
  • bf16: True
  • dataloader_num_workers: 2

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 16
  • num_train_epochs: 1
  • max_steps: -1
  • learning_rate: 2e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.1
  • optim: adamw_torch
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 4
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 8
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 2
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • warmup_ratio: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Click to expand
Epoch Step Training Loss
0.0099 200 679.9589
0.0197 400 1.7857
0.0296 600 1.0238
0.0394 800 0.8982
0.0493 1000 0.8221
0.0592 1200 0.7875
0.0690 1400 0.7405
0.0789 1600 0.7131
0.0887 1800 0.6761
0.0986 2000 0.6732
0.1085 2200 0.6397
0.1183 2400 0.6303
0.1282 2600 0.6021
0.1380 2800 0.5794
0.1479 3000 0.5552
0.1578 3200 0.5638
0.1676 3400 0.5633
0.1775 3600 0.5550
0.1873 3800 0.5473
0.1972 4000 0.5281
0.2070 4200 0.5189
0.2169 4400 0.5282
0.2268 4600 0.5080
0.2366 4800 0.5046
0.2465 5000 0.5111
0.2563 5200 0.4957
0.2662 5400 0.4932
0.2761 5600 0.5045
0.2859 5800 0.5019
0.2958 6000 0.4789
0.3056 6200 0.4866
0.3155 6400 0.4810
0.3254 6600 0.4891
0.3352 6800 0.4800
0.3451 7000 0.4659
0.3549 7200 0.4612
0.3648 7400 0.4644
0.3747 7600 0.4364
0.3845 7800 0.4672
0.3944 8000 0.4546
0.4042 8200 0.4631
0.4141 8400 0.4565
0.4240 8600 0.4242
0.4338 8800 0.4425
0.4437 9000 0.4597
0.4535 9200 0.4292
0.4634 9400 0.4406
0.4733 9600 0.4331
0.4831 9800 0.4342
0.4930 10000 0.4483
0.5028 10200 0.4254
0.5127 10400 0.4182
0.5226 10600 0.4173
0.5324 10800 0.4120
0.5423 11000 0.4197
0.5521 11200 0.4029
0.5620 11400 0.4016
0.5719 11600 0.4159
0.5817 11800 0.4033
0.5916 12000 0.4250
0.6014 12200 0.4013
0.6113 12400 0.4234
0.6211 12600 0.4103
0.6310 12800 0.4046
0.6409 13000 0.4123
0.6507 13200 0.4074
0.6606 13400 0.4098
0.6704 13600 0.4192
0.6803 13800 0.3887
0.6902 14000 0.4017
0.7000 14200 0.3956
0.7099 14400 0.3971
0.7197 14600 0.3964
0.7296 14800 0.3921
0.7395 15000 0.3947
0.7493 15200 0.3795
0.7592 15400 0.3813
0.7690 15600 0.3931
0.7789 15800 0.3792
0.7888 16000 0.3878
0.7986 16200 0.3703
0.8085 16400 0.3836
0.8183 16600 0.3871
0.8282 16800 0.3751
0.8381 17000 0.4005
0.8479 17200 0.3887
0.8578 17400 0.3873
0.8676 17600 0.3811
0.8775 17800 0.3841
0.8874 18000 0.3841
0.8972 18200 0.3701
0.9071 18400 0.3913
0.9169 18600 0.3848
0.9268 18800 0.3885
0.9367 19000 0.3648
0.9465 19200 0.3842
0.9564 19400 0.3989
0.9662 19600 0.3930
0.9761 19800 0.3944
0.9860 20000 0.3712
0.9958 20200 0.3762

Training Time

  • Training: 48.5 minutes

Framework Versions

  • Python: 3.11.12
  • Sentence Transformers: 5.6.0
  • Transformers: 5.12.1
  • PyTorch: 2.7.0+cu128
  • Accelerate: 1.14.0
  • Datasets: 5.0.0
  • Tokenizers: 0.22.2

Additional Resources

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss 

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss 

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

FlopsLoss 

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}