Hugging Face's logo

rdxtremity
/
search-stack-v1

Joblib
ONNX
Safetensors
English
Arabic
retrieval
search
lightgbm
cross-encoder
bge-m3
Model card Files
xet
 Community
search-stack-v1 / splade_v4 /README.md
rdxtremity's picture
rdxtremity
add fine-tuned SPLADE v4 (occasion fix)
a93040e verified
preview code
|
Raw
History Blame Contribute Delete
19.5 kB
metadata 
tags:
  - sentence-transformers
  - sparse-encoder
  - sparse
  - splade
  - generated_from_trainer
  - dataset_size:633174
  - loss:SpladeLoss
  - loss:SparseMultipleNegativesRankingLoss
  - loss:FlopsLoss
base_model: distilbert/distilbert-base-multilingual-cased
widget:
  - text: >-
      [Styli] Wired Floral Lace Balconette Bra Pink (Pink) | حمالة صدر بالكونيت
      دانتيل بتطريز زهري وحواف مقوسة بدون حشوة وسلكية مع لوحة مقصوصة مخيطة
      (وردي). Category: Lingerie & Underwear > Bras.
  - text: >-
      [SHAPES] OVERSIZED COMFORT HOODIE (Black) | هودي مريح كبير الحجم (أسود).
      Category:  > .
  - text: >-
      [Splash] Linen Blend Polo T-shirt Blue (Blue) | Linen Blend Polo T-shirt
      (أزرق). Category: Tops > Shirts & Button-Downs.
  - text: >-
      [CAMPUS] Women's Pastel Low Top Sneakers Beige (Beige) | أحذية رياضية
      نسائية قصيرة بتدرجات الباستيل البارزة (بيج). Category: Shoes > Sneakers.
  - text: >-
      [Karl Lagerfeld] Women's Long Sleeve Printed Top Black (Black) | بلوزة
      مطبوعة بالكامل بأكمام طويلة ورقبة دائرية للنساء، متعددة الألوان (أسود).
      Category:  > .
pipeline_tag: feature-extraction
library_name: sentence-transformers

SPLADE Sparse Encoder

This is a SPLADE Sparse Encoder model finetuned from distilbert/distilbert-base-multilingual-cased on the pairs and triplets datasets using the sentence-transformers library. It maps sentences & paragraphs to a 119547-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

  • Model Type: SPLADE Sparse Encoder
  • Base model: distilbert/distilbert-base-multilingual-cased
  • Maximum Sequence Length: 128 tokens
  • Output Dimensionality: 119547 dimensions
  • Similarity Function: Dot Product
  • Supported Modality: Text
  • Training Datasets:
    • pairs
    • triplets

Model Sources

Full Model Architecture 

SparseEncoder(
  (0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 119547})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("sparse_encoder_model_id")
# Run inference
sentences = [
    'كندرة طولية مطبوعة رجالي',
    "[Karl Lagerfeld] Women's Long Sleeve Printed Top Black (Black) | بلوزة مطبوعة بالكامل بأكمام طويلة ورقبة دائرية للنساء، متعددة الألوان (أسود). Category:  > .",
    '[Splash Fav] Regular Fit Twill Blazer Cream (Cream) | Regular Fit Twill Blazer with Button Closure (كريمي). Category: Suits > Blazers.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 119547]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 30.1986,  30.1103,  15.3038],
#         [ 30.1103, 193.7718,  59.9433],
#         [ 15.3038,  59.9433, 248.5168]])

Training Details

Training Datasets

pairs

  • Dataset: pairs
  • Size: 605,436 training samples
  • Columns: query and positive
  • Approximate statistics based on the first 100 samples:
    query positive
    type string string
    modality text text
    details
    • min: 6 tokens
    • mean: 10.33 tokens
    • max: 20 tokens
    • min: 28 tokens
    • mean: 51.14 tokens
    • max: 124 tokens
  • Samples:
    query positive
    men's watch analog [CASIO] Leather Strap Analog Watch (Mint Green) | ساعة انالوج جلد اصلي (أخضر نعناعي). Category: > .
    black hoodie women [Tribe of 6] Women's Logo Hooded Sweatshirt Black (Black) | سويت شيرت نسائي سادة بأكمام طويلة وغطاء رأس، أسود (أسود). Category: Hoodies & Sweatshirt > Hoodies.
    وشاح مخطط للنساء [MANGO] Geometric Stripe Scarf Beige (Beige) | وشاح بنقوش هندسية مخططة (بيج). Category: Accessories > Scarves.
  • Loss: SpladeLoss with these parameters:
    {
    "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
    "document_regularizer_weight": 3e-05,
    "query_regularizer_weight": 5e-05
}

triplets

  • Dataset: triplets
  • Size: 27,738 training samples
  • Columns: query, positive, and negative
  • Approximate statistics based on the first 100 samples:
    query positive negative
    type string string string
    modality text text text
    details
    • min: 11 tokens
    • mean: 12.15 tokens
    • max: 14 tokens
    • min: 34 tokens
    • mean: 48.58 tokens
    • max: 61 tokens
    • min: 48 tokens
    • mean: 52.9 tokens
    • max: 62 tokens
  • Samples:
    query positive negative
    كندرة أونلاين بسروات زين [Khizana] Denim A-line Skirt Steel Blue (Steel Blue) | تنورة - من الدنيم (أزرق فولاذي). Category: Denim > Denim Skirts. [Tchibo] Women's Checkered Pull-On Shorts Dark Blue (Dark Blue) | شورت بنمط مربعات سهل الارتداء للنساء، أزرق (أزرق غامق). Category: > .
    كندرة أونلاين بسروات زين [Khizana] Denim A-line Skirt Steel Blue (Steel Blue) | تنورة - من الدنيم (أزرق فولاذي). Category: Denim > Denim Skirts. [Styli] Styli Set of 2 Ribbon Detail Socks Light Steel Blue (Light Steel Blue) | طقم جوارب من قطعتين بتفاصيل شريط (أزرق فولاذي فاتح). Category: Shoes > Socks.
    كندرة أونلاين بسروات زين [Khizana] Denim A-line Skirt Steel Blue (Steel Blue) | تنورة - من الدنيم (أزرق فولاذي). Category: Denim > Denim Skirts. [Tchibo] Women's Checkered Pull-On Shorts Dark Blue (Dark Blue) | شورت بنمط مربعات سهل الارتداء للنساء، أزرق (أزرق غامق). Category: > .
  • Loss: SpladeLoss with these parameters:
    {
    "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
    "document_regularizer_weight": 3e-05,
    "query_regularizer_weight": 5e-05
}

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 16
  • num_train_epochs: 1
  • learning_rate: 2e-05
  • warmup_steps: 0.1
  • gradient_accumulation_steps: 4
  • bf16: True
  • dataloader_num_workers: 2

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 16
  • num_train_epochs: 1
  • max_steps: -1
  • learning_rate: 2e-05
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.1
  • optim: adamw_torch
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 4
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 8
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 2
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • warmup_ratio: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss
0.0202 200 518.0875
0.0404 400 1.3120
0.0606 600 1.0065
0.0809 800 0.9330
0.1011 1000 0.8807
0.1213 1200 0.8054
0.1415 1400 0.7627
0.1617 1600 0.7404
0.1819 1800 0.7277
0.2022 2000 0.7013
0.2224 2200 0.6515
0.2426 2400 0.6359
0.2628 2600 0.6334
0.2830 2800 0.6284
0.3032 3000 0.6302
0.3234 3200 0.5996
0.3437 3400 0.5933
0.3639 3600 0.5846
0.3841 3800 0.5783
0.4043 4000 0.5907
0.4245 4200 0.5669
0.4447 4400 0.5768
0.4650 4600 0.5507
0.4852 4800 0.5403
0.5054 5000 0.5372
0.5256 5200 0.5382
0.5458 5400 0.5308
0.5660 5600 0.5453
0.5862 5800 0.5218
0.6065 6000 0.5058
0.6267 6200 0.5229
0.6469 6400 0.5245
0.6671 6600 0.5115
0.6873 6800 0.5023
0.7075 7000 0.5049
0.7278 7200 0.5155
0.7480 7400 0.4976
0.7682 7600 0.4965
0.7884 7800 0.4968
0.8086 8000 0.4950
0.8288 8200 0.4889
0.8490 8400 0.4918
0.8693 8600 0.4870
0.8895 8800 0.4723
0.9097 9000 0.4808
0.9299 9200 0.4791
0.9501 9400 0.4829
0.9703 9600 0.4654
0.9905 9800 0.4931

Training Time

  • Training: 37.1 minutes

Framework Versions

  • Python: 3.11.12
  • Sentence Transformers: 5.6.0
  • Transformers: 5.12.1
  • PyTorch: 2.7.0+cu128
  • Accelerate: 1.14.0
  • Datasets: 5.0.0
  • Tokenizers: 0.22.2

Additional Resources

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss 

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss 

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

FlopsLoss 

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}