Add new SparseEncoder model

1_SpladePooling/config.json

@@ -0,0 +1,5 @@
+{
+    "pooling_strategy": "max",
+    "activation_function": "relu",
+    "word_embedding_dimension": 30522
+}

README.md

@@ -0,0 +1,516 @@
+---
+language:
+- en
+license: mit
+tags:
+- sentence-transformers
+- sparse-encoder
+- sparse
+- splade
+- generated_from_trainer
+- dataset_size:250000
+- loss:SpladeLoss
+- loss:SparseMultipleNegativesRankingLoss
+- loss:FlopsLoss
+base_model: prajjwal1/bert-mini
+widget:
+- text: icd medication reaction
+- text: Report Abuse. An egg lives for around 12 hours after ovulation. Sperm can
+    live for about five days inside the uterus, so providing those two time frames
+    collide, it can be pretty soon after sex. Hours, usually. Implantation occurs
+    between 6 - 12 days.
+- text: 'A warm-up is important for many reasons. Some of these reasons include: -
+    Facilitates transition from rest to exercise-Stretches postural muscles-Augments
+    blood flow-Ele … vates body temperature-Allows body to adjust to changing physiologic,
+    biomechanical and bioenergetic demands placed on it during conditioning phase.
+    warm-up helps your body prepare for any physical activity. Without a warm-up,
+    your muscles will be cold and stiff, the oxygen won''t be flowing to your muscles
+    and joints and you will not perform the activity well. Also, when you do a warm
+    up your recovery from exercising will be more comfortable and shorter.'
+- text: First, you need to have a Kindle Fire HD or HDX as these are the Kindle Fires
+    that have bluetooth. The very first generation doesn't have this capability. (If
+    you don't know which tablet you have, see this article.). Second, you need to
+    have a bluetooth keyboard or other device, like headphones, speakers, or earbuds.
+    This is a picture of a Jawbone earpiece I've successfully paired to my Kindle
+    Fire and been able to listen to music with.
+- text: Cantigny Park. Cantigny is a 500-acre (2.0 km2) park in Wheaton, Illinois,
+    30 miles west of Chicago. It is the former estate of Joseph Medill and his grandson
+    Colonel Robert R. McCormick, publishers of the Chicago Tribune, and is open to
+    the public.
+pipeline_tag: feature-extraction
+library_name: sentence-transformers
+metrics:
+- dot_accuracy@1
+- dot_accuracy@3
+- dot_accuracy@5
+- dot_accuracy@10
+- dot_precision@1
+- dot_precision@3
+- dot_precision@5
+- dot_precision@10
+- dot_recall@1
+- dot_recall@3
+- dot_recall@5
+- dot_recall@10
+- dot_ndcg@10
+- dot_mrr@10
+- dot_map@100
+- query_active_dims
+- query_sparsity_ratio
+- corpus_active_dims
+- corpus_sparsity_ratio
+model-index:
+- name: SPLADE-BERT-Mini
+  results:
+  - task:
+      type: sparse-information-retrieval
+      name: Sparse Information Retrieval
+    dataset:
+      name: Unknown
+      type: unknown
+    metrics:
+    - type: dot_accuracy@1
+      value: 0.63028
+      name: Dot Accuracy@1
+    - type: dot_accuracy@3
+      value: 0.79716
+      name: Dot Accuracy@3
+    - type: dot_accuracy@5
+      value: 0.85096
+      name: Dot Accuracy@5
+    - type: dot_accuracy@10
+      value: 0.90548
+      name: Dot Accuracy@10
+    - type: dot_precision@1
+      value: 0.63028
+      name: Dot Precision@1
+    - type: dot_precision@3
+      value: 0.26571999999999996
+      name: Dot Precision@3
+    - type: dot_precision@5
+      value: 0.170192
+      name: Dot Precision@5
+    - type: dot_precision@10
+      value: 0.09054800000000002
+      name: Dot Precision@10
+    - type: dot_recall@1
+      value: 0.63028
+      name: Dot Recall@1
+    - type: dot_recall@3
+      value: 0.79716
+      name: Dot Recall@3
+    - type: dot_recall@5
+      value: 0.85096
+      name: Dot Recall@5
+    - type: dot_recall@10
+      value: 0.90548
+      name: Dot Recall@10
+    - type: dot_ndcg@10
+      value: 0.7689713558276354
+      name: Dot Ndcg@10
+    - type: dot_mrr@10
+      value: 0.7250807142857304
+      name: Dot Mrr@10
+    - type: dot_map@100
+      value: 0.728785316630622
+      name: Dot Map@100
+    - type: query_active_dims
+      value: 26.435239791870117
+      name: Query Active Dims
+    - type: query_sparsity_ratio
+      value: 0.9991338955575693
+      name: Query Sparsity Ratio
+    - type: corpus_active_dims
+      value: 326.6760399121094
+      name: Corpus Active Dims
+    - type: corpus_sparsity_ratio
+      value: 0.9892970303416517
+      name: Corpus Sparsity Ratio
+---
+
+# SPLADE-BERT-Mini
+
+This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [prajjwal1/bert-mini](https://huggingface.co/prajjwal1/bert-mini) using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space   and can be used for semantic search and sparse retrieval.
+## Model Details
+
+### Model Description
+- **Model Type:** SPLADE Sparse Encoder
+- **Base model:** [prajjwal1/bert-mini](https://huggingface.co/prajjwal1/bert-mini) <!-- at revision 5e123abc2480f0c4b4cac186d3b3f09299c258fc -->
+- **Maximum Sequence Length:** 512 tokens
+- **Output Dimensionality:** 30522 dimensions
+- **Similarity Function:** Dot Product
+<!-- - **Training Dataset:** Unknown -->
+- **Language:** en
+- **License:** mit
+
+### Model Sources
+
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder)
+
+### Full Model Architecture
+
+```
+SparseEncoder(
+  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
+  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
+)
+```
+
+## Usage
+
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
+
+```bash
+pip install -U sentence-transformers
+```
+
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SparseEncoder
+
+# Download from the 🤗 Hub
+model = SparseEncoder("rasyosef/SPLADE-BERT-Mini")
+# Run inference
+queries = [
+    "cantigny gardens cost",
+]
+documents = [
+    'The fee for a ceremony ranges from $400 to $2,500 with reception rental or $3,000 for a ceremony-only wedding. Please inquire about discounted rates for ceremony guest counts under 75. The average wedding cost at Cantigny Park is estimated at between $12,881 and $22,238 for a ceremony & reception for 100 guests.',
+    'Nestled in a serene setting, Cantigny Park is a scenic realm where you will create a unique wedding, the memories of which you will always cherish. This expansive estate encompasses 500 acres of beautiful gardens, colorful botanicals and tranquil water features, creating an idyllic background for this ideal day.',
+    'Cantigny Park. Cantigny is a 500-acre (2.0 km2) park in Wheaton, Illinois, 30 miles west of Chicago. It is the former estate of Joseph Medill and his grandson Colonel Robert R. McCormick, publishers of the Chicago Tribune, and is open to the public.',
+]
+query_embeddings = model.encode_query(queries)
+document_embeddings = model.encode_document(documents)
+print(query_embeddings.shape, document_embeddings.shape)
+# [1, 30522] [3, 30522]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(query_embeddings, document_embeddings)
+print(similarities)
+# tensor([[18.8703, 13.8253, 13.4587]])
+```
+
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+## Evaluation
+
+### Metrics
+
+#### Sparse Information Retrieval
+
+* Evaluated with [<code>SparseInformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator)
+
+| Metric                | Value     |
+|:----------------------|:----------|
+| dot_accuracy@1        | 0.6303    |
+| dot_accuracy@3        | 0.7972    |
+| dot_accuracy@5        | 0.851     |
+| dot_accuracy@10       | 0.9055    |
+| dot_precision@1       | 0.6303    |
+| dot_precision@3       | 0.2657    |
+| dot_precision@5       | 0.1702    |
+| dot_precision@10      | 0.0905    |
+| dot_recall@1          | 0.6303    |
+| dot_recall@3          | 0.7972    |
+| dot_recall@5          | 0.851     |
+| dot_recall@10         | 0.9055    |
+| **dot_ndcg@10**       | **0.769** |
+| dot_mrr@10            | 0.7251    |
+| dot_map@100           | 0.7288    |
+| query_active_dims     | 26.4352   |
+| query_sparsity_ratio  | 0.9991    |
+| corpus_active_dims    | 326.676   |
+| corpus_sparsity_ratio | 0.9893    |
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Dataset
+
+#### Unnamed Dataset
+
+* Size: 250,000 training samples
+* Columns: <code>query</code>, <code>positive</code>, <code>negative_1</code>, and <code>negative_2</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | query                                                                            | positive                                                                            | negative_1                                                                          | negative_2                                                                          |
+  |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
+  | type    | string                                                                           | string                                                                              | string                                                                              | string                                                                              |
+  | details | <ul><li>min: 4 tokens</li><li>mean: 8.87 tokens</li><li>max: 31 tokens</li></ul> | <ul><li>min: 20 tokens</li><li>mean: 82.54 tokens</li><li>max: 218 tokens</li></ul> | <ul><li>min: 20 tokens</li><li>mean: 79.98 tokens</li><li>max: 252 tokens</li></ul> | <ul><li>min: 19 tokens</li><li>mean: 80.55 tokens</li><li>max: 211 tokens</li></ul> |
+* Samples:
+  | query                                                                   | positive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | negative_1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | negative_2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+  |:------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>how do automotive technicians get paid</code>                     | <code>104 months ago. The amount of pay from company to company does not vary too much, but you do have a wide variety of compensation methods. There are various combinations of hourly and commission pay rates, which depending on what type of work you specialize in can vary your bottom line considerably.04 months ago. The amount of pay from company to company does not vary too much, but you do have a wide variety of compensation methods. There are various combinations of hourly and commission pay rates, which depending on what type of work you specialize in can vary your bottom line considerably.</code> | <code>Bureau of Labor Statistics figures indicate that automotive technicians earned an average annual salary of $38,560 and an average hourly wage of $18.54 as of May 2011.Half of auto technicians reported annual salaries of between $26,850 and $47,540 and hourly wages of between $12.91 and $22.86.The 10 percent of automotive techs who earned the lowest made $20,620 or less per year, and the top 10 percent of earners made $59,600 or more per year.ver one-third of all automotive technicians employed as of May 2011 worked in the automotive repair and maintenance industry, where they earned an average of $35,090 per year.</code> | <code>It really depends on what automaker your working for, how much experience you have, and how long you've been in the industry. Obviously if you're working for a highend company(BMW,Mercedes,Ferrari) you can expect to be paid more per hour. And automotive technicians don't get paid by the hour.We get paid per FLAT RATE hour. Which basically means that we get paid by the job. Which could range from 0.2 of an hour for replacing a headlight bulb to 10hours for a transmission overhaul. Then there's a difference between warranty jobs and cash jobs.ut I won't get into too much detail. Automotive technicians get paid around $12-$15/hr at entry level. But can make around $18-$26/hr with much more experience. Which means you can expect to make 30,000 to 60,000/year. Though most technicians don't see past 45,000 a year.</code> |
+  | <code>how far is steamboat springs from golden?</code>                  | <code>The distance between Steamboat Springs and Golden in a straight line is 100 miles or 160.9 Kilometers. Driving Directions & Drive Times from Steamboat Springs to Golden can be found further down the page.</code>                                                                                                                                                                                                                                                                                                                                                                                                          | <code>Steamboat Springs Vacation Rentals Steamboat Springs Vacations Steamboat Springs Restaurants Things to Do in Steamboat Springs Steamboat Springs Travel Forum Steamboat Springs Photos Steamboat Springs Map Steamboat Springs Travel Guide All Steamboat Springs Hotels; Steamboat Springs Hotel Deals; Last Minute Hotels in Steamboat Springs; By Hotel Type Steamboat Springs Family Hotels</code>                                                                                                                                                                                                                                               | <code>There are 98.92 miles from Golden to Steamboat Springs in northwest direction and 143 miles (230.14 kilometers) by car, following the US-40 route. Golden and Steamboat Springs are 3 hours 20 mins far apart, if you drive non-stop. This is the fastest route from Golden, CO to Steamboat Springs, CO. The halfway point is Heeney, CO. Golden, CO and Steamboat Springs, CO are in the same time zone (MDT). Current time in both locations is 1:26 pm.</code>                                                                                                                                                                                                                                                                                                                                                                                         |
+  | <code>incoming wire routing number for california bank and trust</code> | <code>Please call California Bank And Trust representative at (888) 315-2271 for more information. 1  Routing Number: 122003396. 2  250 EAST FIRST STREET # 700. LOS ANGELES, CA 90012-0000. 3  Phone Number: (888) 315-2271.</code>                                                                                                                                                                                                                                                                                                                                                                                               | <code>When asked to provide a routing number for incoming wire transfers to Union Bank accounts, the routing number to use is: 122000496. back to top What options do I have to send wires?</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                         | <code>Business Contracting Officers (BCO) have access to Online Banking wires. Simply sign on to Online Banking, click “Send Wires”, and then complete the required information. This particular service is limited to sending wires to U.S. banks only.</code>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
+* Loss: [<code>SpladeLoss</code>](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters:
+  ```json
+  {
+      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
+      "document_regularizer_weight": 0.001,
+      "query_regularizer_weight": 0.002
+  }
+  ```
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `eval_strategy`: epoch
+- `per_device_train_batch_size`: 64
+- `per_device_eval_batch_size`: 64
+- `learning_rate`: 6e-05
+- `num_train_epochs`: 4
+- `lr_scheduler_type`: cosine
+- `warmup_ratio`: 0.025
+- `fp16`: True
+- `optim`: adamw_torch_fused
+- `batch_sampler`: no_duplicates
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: epoch
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 64
+- `per_device_eval_batch_size`: 64
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `torch_empty_cache_steps`: None
+- `learning_rate`: 6e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1.0
+- `num_train_epochs`: 4
+- `max_steps`: -1
+- `lr_scheduler_type`: cosine
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.025
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: False
+- `fp16`: True
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch_fused
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: None
+- `hub_always_push`: False
+- `hub_revision`: None
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `include_for_metrics`: []
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `use_liger_kernel`: False
+- `liger_kernel_config`: None
+- `eval_use_gather_object`: False
+- `average_tokens_across_devices`: False
+- `prompts`: None
+- `batch_sampler`: no_duplicates
+- `multi_dataset_batch_sampler`: proportional
+- `router_mapping`: {}
+- `learning_rate_mapping`: {}
+
+</details>
+
+### Training Logs
+| Epoch | Step  | Training Loss | dot_ndcg@10 |
+|:-----:|:-----:|:-------------:|:-----------:|
+| 1.0   | 3907  | 23.8846       | 0.7509      |
+| 2.0   | 7814  | 0.785         | 0.7670      |
+| 3.0   | 11721 | 0.6873        | 0.7685      |
+| 4.0   | 15628 | 0.6283        | 0.7690      |
+| -1    | -1    | -             | 0.7690      |
+
+
+### Framework Versions
+- Python: 3.11.13
+- Sentence Transformers: 5.0.0
+- Transformers: 4.53.1
+- PyTorch: 2.6.0+cu124
+- Accelerate: 1.8.1
+- Datasets: 3.6.0
+- Tokenizers: 0.21.2
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+
+#### SpladeLoss
+```bibtex
+@misc{formal2022distillationhardnegativesampling,
+      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
+      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
+      year={2022},
+      eprint={2205.04733},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR},
+      url={https://arxiv.org/abs/2205.04733},
+}
+```
+
+#### SparseMultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+
+#### FlopsLoss
+```bibtex
+@article{paria2020minimizing,
+    title={Minimizing flops to learn efficient sparse representations},
+    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
+    journal={arXiv preprint arXiv:2004.05665},
+    year={2020}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+--> 

config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 4,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.53.1",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SparseEncoder",
+  "__version__": {
+    "sentence_transformers": "5.0.0",
+    "transformers": "4.53.1",
+    "pytorch": "2.6.0+cu124"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "dot"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:02ac0b450c721891da6146541abdcbb030bc4969b0f9817c3a9d4073a720241a
+size 44814856

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.sparse_encoder.models.MLMTransformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_SpladePooling",
+    "type": "sentence_transformers.sparse_encoder.models.SpladePooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 512,
+    "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}