rdxtremity's picture
add SPLADE v6_v2 (qwen30-pairs A/B winner, sparse nDCG 0.8189)
b737adb verified
|
Raw
History Blame Contribute Delete
21.7 kB
---
tags:
- sentence-transformers
- sparse-encoder
- sparse
- splade
- generated_from_trainer
- dataset_size:1298215
- loss:SpladeLoss
- loss:SparseMultipleNegativesRankingLoss
- loss:FlopsLoss
base_model: distilbert/distilbert-base-multilingual-cased
widget:
- text: '[Clan London] Women''s Logo Tote Bag Olive (Olive) | حقيبة توت بشعار العلامة
التجارية للنساء، زيتي (زيتي). Category: Bags > Shoppers & Totes.'
- text: '[Fire & Glory] Women''s Animal Print Midi Dress Light Gray (Light Gray) |
فستان ميدي بأكمام طويلة وطبعة جرافيك للنساء، أسود (رمادي فاتح). Category: Dresses
> Midi Dresses.'
- text: '[melissa] Melissa Becky AD Pink (Pink) | ميليسا بيكي إيه دي (وردي). Category:
Shoes > Slides & Flip-Flops.'
- text: '[H&M] Curvy Ultra High-Rise Wide-Leg Jeans (Pastel Blue) | جينز كيرفي بخصر
مرتفع جداً وأرجل واسعة (أزرق باستيل). Category: Denim > Jeans.'
- text: '[Splash Basics] Flexi Comfort Drawstring A-line Midi Skirt Blue (Blue) |
الرباط ألف خط تنورة ميدي (أزرق). Category: Bottoms > Skirts.'
pipeline_tag: feature-extraction
library_name: sentence-transformers
---
# SPLADE Sparse Encoder
This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [distilbert/distilbert-base-multilingual-cased](https://huggingface.co/distilbert/distilbert-base-multilingual-cased) on the pairs and triplets datasets using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 119547-dimensional sparse vector space and can be used for semantic search and sparse retrieval.
## Model Details
### Model Description
- **Model Type:** SPLADE Sparse Encoder
- **Base model:** [distilbert/distilbert-base-multilingual-cased](https://huggingface.co/distilbert/distilbert-base-multilingual-cased) <!-- at revision 45c032ab32cc946ad88a166f7cb282f58c753c2e -->
- **Maximum Sequence Length:** 128 tokens
- **Output Dimensionality:** 119547 dimensions
- **Similarity Function:** Dot Product
- **Supported Modality:** Text
- **Training Datasets:**
- pairs
- triplets
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
- **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder)
### Full Model Architecture
```
SparseEncoder(
(0): Transformer({'transformer_task': 'fill-mask', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'token_embeddings', 'architecture': 'DistilBertForMaskedLM'})
(1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'embedding_dimension': 119547})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import SparseEncoder
# Download from the 🤗 Hub
model = SparseEncoder("sparse_encoder_model_id")
# Run inference
sentences = [
'شنطة زهّانة',
'[PANDORA] Aquarius Zodiac Charm (Silver) | دلاية برج الدلو (فضي). Category: > Accessories.',
"[Hurley] Women's Crew Neck Solid Sweatshirt Dark Salmon (Dark Salmon) | سويت شيرت نسائي سادة بياقة دائرية، برتقالي (سلموني غامق). Category: Hoodies & Sweatshirt > Sweatshirts.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 119547]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 11.9930, 13.0235, 7.8521],
# [ 13.0235, 491.0651, 65.4858],
# [ 7.8521, 65.4858, 246.8788]])
```
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
<!--
## Bias, Risks and Limitations
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->
<!--
### Recommendations
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->
## Training Details
### Training Datasets
#### pairs
* Dataset: pairs
* Size: 1,292,307 training samples
* Columns: <code>query</code> and <code>positive</code>
* Approximate statistics based on the first 100 samples:
| | query | positive |
|:---------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
| type | string | string |
| modality | text | text |
| details | <ul><li>min: 6 tokens</li><li>mean: 10.33 tokens</li><li>max: 20 tokens</li></ul> | <ul><li>min: 28 tokens</li><li>mean: 51.14 tokens</li><li>max: 124 tokens</li></ul> |
* Samples:
| query | positive |
|:--------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>men's watch analog</code> | <code>[CASIO] Leather Strap Analog Watch (Mint Green) \| ساعة انالوج جلد اصلي (أخضر نعناعي). Category: > .</code> |
| <code>black hoodie women</code> | <code>[Tribe of 6] Women's Logo Hooded Sweatshirt Black (Black) \| سويت شيرت نسائي سادة بأكمام طويلة وغطاء رأس، أسود (أسود). Category: Hoodies & Sweatshirt > Hoodies.</code> |
| <code>وشاح مخطط للنساء</code> | <code>[MANGO] Geometric Stripe Scarf Beige (Beige) \| وشاح بنقوش هندسية مخططة (بيج). Category: Accessories > Scarves.</code> |
* Loss: [<code>SpladeLoss</code>](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters:
```json
{
"loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
"document_regularizer_weight": 3e-05,
"query_regularizer_weight": 5e-05
}
```
#### triplets
* Dataset: triplets
* Size: 5,908 training samples
* Columns: <code>query</code>, <code>positive</code>, and <code>negative</code>
* Approximate statistics based on the first 100 samples:
| | query | positive | negative |
|:---------|:---------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|
| type | string | string | string |
| modality | text | text | text |
| details | <ul><li>min: 5 tokens</li><li>mean: 8.75 tokens</li><li>max: 17 tokens</li></ul> | <ul><li>min: 39 tokens</li><li>mean: 57.33 tokens</li><li>max: 82 tokens</li></ul> | <ul><li>min: 38 tokens</li><li>mean: 54.78 tokens</li><li>max: 88 tokens</li></ul> |
* Samples:
| query | positive | negative |
|:----------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <code>don't call me jennyfer jumpsuits & bodysuits</code> | <code>[Don't Call Me Jennyfer] Women's One Sleeve Bodysuit Green (Green) \| بودي سوت سادة بأكمام قصيرة للنساء، أخضر (أخضر). Category: Jumpsuits & Bodysuits > Body Suits.</code> | <code>[Ashita Fernandes] Kenza Cupro V-Neck Jumpsuit Royal Blue (Facebook Blue) \| كينزا جمبسوت كوبرو بفتحة رقبة V أزرق ملكي (أزرق فيسبوك). Category: Jumpsuits & Bodysuits > Jumpsuits.</code> |
| <code>don't call me jennyfer jumpsuits & bodysuits</code> | <code>[Don't Call Me Jennyfer] Women's One Sleeve Bodysuit Green (Green) \| بودي سوت سادة بأكمام قصيرة للنساء، أخضر (أخضر). Category: Jumpsuits & Bodysuits > Body Suits.</code> | <code>[Splash Intimates] Mesh Panel Shaping Bodysuit (Black) \| بودي سوت مشد بأجزاء شبكية (أسود). Category: Jumpsuits & Bodysuits > Body Suits.</code> |
| <code>don't call me jennyfer jumpsuits & bodysuits</code> | <code>[Don't Call Me Jennyfer] Women's One Sleeve Bodysuit Green (Green) \| بودي سوت سادة بأكمام قصيرة للنساء، أخضر (أخضر). Category: Jumpsuits & Bodysuits > Body Suits.</code> | <code>[Ashita Fernandes] Kenza Cupro V-Neck Jumpsuit Royal Blue (Facebook Blue) \| كينزا جمبسوت كوبرو بفتحة رقبة V أزرق ملكي (أزرق فيسبوك). Category: Jumpsuits & Bodysuits > Jumpsuits.</code> |
* Loss: [<code>SpladeLoss</code>](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters:
```json
{
"loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False, directions=('query_to_doc',), partition_mode='joint', hardness_mode=None, hardness_strength=0.0)",
"document_regularizer_weight": 3e-05,
"query_regularizer_weight": 5e-05
}
```
### Training Hyperparameters
#### Non-Default Hyperparameters
- `per_device_train_batch_size`: 16
- `num_train_epochs`: 1
- `learning_rate`: 2e-05
- `warmup_steps`: 0.1
- `gradient_accumulation_steps`: 4
- `bf16`: True
- `dataloader_num_workers`: 2
#### All Hyperparameters
<details><summary>Click to expand</summary>
- `per_device_train_batch_size`: 16
- `num_train_epochs`: 1
- `max_steps`: -1
- `learning_rate`: 2e-05
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: None
- `warmup_steps`: 0.1
- `optim`: adamw_torch
- `optim_args`: None
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `optim_target_modules`: None
- `gradient_accumulation_steps`: 4
- `average_tokens_across_devices`: True
- `max_grad_norm`: 1.0
- `label_smoothing_factor`: 0.0
- `bf16`: True
- `fp16`: False
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `use_liger_kernel`: False
- `liger_kernel_config`: None
- `use_cache`: False
- `neftune_noise_alpha`: None
- `torch_empty_cache_steps`: None
- `auto_find_batch_size`: False
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `include_num_input_tokens_seen`: no
- `log_level`: passive
- `log_level_replica`: warning
- `disable_tqdm`: False
- `project`: huggingface
- `trackio_space_id`: None
- `trackio_bucket_id`: None
- `trackio_static_space_id`: None
- `per_device_eval_batch_size`: 8
- `prediction_loss_only`: True
- `eval_on_start`: False
- `eval_do_concat_batches`: True
- `eval_use_gather_object`: False
- `eval_accumulation_steps`: None
- `include_for_metrics`: []
- `batch_eval_metrics`: False
- `save_only_model`: False
- `save_on_each_node`: False
- `enable_jit_checkpoint`: False
- `push_to_hub`: False
- `hub_private_repo`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_always_push`: False
- `hub_revision`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `restore_callback_states_from_checkpoint`: False
- `full_determinism`: False
- `seed`: 42
- `data_seed`: None
- `use_cpu`: False
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `parallelism_config`: None
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 2
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `dataloader_prefetch_factor`: None
- `remove_unused_columns`: True
- `label_names`: None
- `train_sampling_strategy`: random
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `ddp_static_graph`: None
- `ddp_backend`: None
- `ddp_timeout`: 1800
- `fsdp`: None
- `fsdp_config`: None
- `deepspeed`: None
- `debug`: []
- `skip_memory_metrics`: True
- `do_predict`: False
- `resume_from_checkpoint`: None
- `warmup_ratio`: None
- `local_rank`: -1
- `prompts`: None
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: proportional
- `router_mapping`: {}
- `learning_rate_mapping`: {}
</details>
### Training Logs
<details><summary>Click to expand</summary>
| Epoch | Step | Training Loss |
|:------:|:-----:|:-------------:|
| 0.0099 | 200 | 679.9589 |
| 0.0197 | 400 | 1.7857 |
| 0.0296 | 600 | 1.0238 |
| 0.0394 | 800 | 0.8982 |
| 0.0493 | 1000 | 0.8221 |
| 0.0592 | 1200 | 0.7875 |
| 0.0690 | 1400 | 0.7405 |
| 0.0789 | 1600 | 0.7131 |
| 0.0887 | 1800 | 0.6761 |
| 0.0986 | 2000 | 0.6732 |
| 0.1085 | 2200 | 0.6397 |
| 0.1183 | 2400 | 0.6303 |
| 0.1282 | 2600 | 0.6021 |
| 0.1380 | 2800 | 0.5794 |
| 0.1479 | 3000 | 0.5552 |
| 0.1578 | 3200 | 0.5638 |
| 0.1676 | 3400 | 0.5633 |
| 0.1775 | 3600 | 0.5550 |
| 0.1873 | 3800 | 0.5473 |
| 0.1972 | 4000 | 0.5281 |
| 0.2070 | 4200 | 0.5189 |
| 0.2169 | 4400 | 0.5282 |
| 0.2268 | 4600 | 0.5080 |
| 0.2366 | 4800 | 0.5046 |
| 0.2465 | 5000 | 0.5111 |
| 0.2563 | 5200 | 0.4957 |
| 0.2662 | 5400 | 0.4932 |
| 0.2761 | 5600 | 0.5045 |
| 0.2859 | 5800 | 0.5019 |
| 0.2958 | 6000 | 0.4789 |
| 0.3056 | 6200 | 0.4866 |
| 0.3155 | 6400 | 0.4810 |
| 0.3254 | 6600 | 0.4891 |
| 0.3352 | 6800 | 0.4800 |
| 0.3451 | 7000 | 0.4659 |
| 0.3549 | 7200 | 0.4612 |
| 0.3648 | 7400 | 0.4644 |
| 0.3747 | 7600 | 0.4364 |
| 0.3845 | 7800 | 0.4672 |
| 0.3944 | 8000 | 0.4546 |
| 0.4042 | 8200 | 0.4631 |
| 0.4141 | 8400 | 0.4565 |
| 0.4240 | 8600 | 0.4242 |
| 0.4338 | 8800 | 0.4425 |
| 0.4437 | 9000 | 0.4597 |
| 0.4535 | 9200 | 0.4292 |
| 0.4634 | 9400 | 0.4406 |
| 0.4733 | 9600 | 0.4331 |
| 0.4831 | 9800 | 0.4342 |
| 0.4930 | 10000 | 0.4483 |
| 0.5028 | 10200 | 0.4254 |
| 0.5127 | 10400 | 0.4182 |
| 0.5226 | 10600 | 0.4173 |
| 0.5324 | 10800 | 0.4120 |
| 0.5423 | 11000 | 0.4197 |
| 0.5521 | 11200 | 0.4029 |
| 0.5620 | 11400 | 0.4016 |
| 0.5719 | 11600 | 0.4159 |
| 0.5817 | 11800 | 0.4033 |
| 0.5916 | 12000 | 0.4250 |
| 0.6014 | 12200 | 0.4013 |
| 0.6113 | 12400 | 0.4234 |
| 0.6211 | 12600 | 0.4103 |
| 0.6310 | 12800 | 0.4046 |
| 0.6409 | 13000 | 0.4123 |
| 0.6507 | 13200 | 0.4074 |
| 0.6606 | 13400 | 0.4098 |
| 0.6704 | 13600 | 0.4192 |
| 0.6803 | 13800 | 0.3887 |
| 0.6902 | 14000 | 0.4017 |
| 0.7000 | 14200 | 0.3956 |
| 0.7099 | 14400 | 0.3971 |
| 0.7197 | 14600 | 0.3964 |
| 0.7296 | 14800 | 0.3921 |
| 0.7395 | 15000 | 0.3947 |
| 0.7493 | 15200 | 0.3795 |
| 0.7592 | 15400 | 0.3813 |
| 0.7690 | 15600 | 0.3931 |
| 0.7789 | 15800 | 0.3792 |
| 0.7888 | 16000 | 0.3878 |
| 0.7986 | 16200 | 0.3703 |
| 0.8085 | 16400 | 0.3836 |
| 0.8183 | 16600 | 0.3871 |
| 0.8282 | 16800 | 0.3751 |
| 0.8381 | 17000 | 0.4005 |
| 0.8479 | 17200 | 0.3887 |
| 0.8578 | 17400 | 0.3873 |
| 0.8676 | 17600 | 0.3811 |
| 0.8775 | 17800 | 0.3841 |
| 0.8874 | 18000 | 0.3841 |
| 0.8972 | 18200 | 0.3701 |
| 0.9071 | 18400 | 0.3913 |
| 0.9169 | 18600 | 0.3848 |
| 0.9268 | 18800 | 0.3885 |
| 0.9367 | 19000 | 0.3648 |
| 0.9465 | 19200 | 0.3842 |
| 0.9564 | 19400 | 0.3989 |
| 0.9662 | 19600 | 0.3930 |
| 0.9761 | 19800 | 0.3944 |
| 0.9860 | 20000 | 0.3712 |
| 0.9958 | 20200 | 0.3762 |
</details>
### Training Time
- **Training**: 48.5 minutes
### Framework Versions
- Python: 3.11.12
- Sentence Transformers: 5.6.0
- Transformers: 5.12.1
- PyTorch: 2.7.0+cu128
- Accelerate: 1.14.0
- Datasets: 5.0.0
- Tokenizers: 0.22.2
## Additional Resources
- [Training and Finetuning Sparse Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sparse-encoder): the end-to-end guide for training or finetuning SPLADE and other sparse encoder models.
## Citation
### BibTeX
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
```
#### SpladeLoss
```bibtex
@misc{formal2022distillationhardnegativesampling,
title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
year={2022},
eprint={2205.04733},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2205.04733},
}
```
#### SparseMultipleNegativesRankingLoss
```bibtex
@misc{oord2019representationlearningcontrastivepredictive,
title={Representation Learning with Contrastive Predictive Coding},
author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
year={2019},
eprint={1807.03748},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/1807.03748},
}
```
#### FlopsLoss
```bibtex
@article{paria2020minimizing,
title={Minimizing flops to learn efficient sparse representations},
author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
journal={arXiv preprint arXiv:2004.05665},
year={2020}
}
```
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->