Add new SparseEncoder model

1_SpladePooling/config.json

@@ -0,0 +1,5 @@
+{
+    "pooling_strategy": "max",
+    "activation_function": "relu",
+    "word_embedding_dimension": 30522
+}

README.md

@@ -0,0 +1,509 @@
+---
+language:
+- en
+license: mit
+tags:
+- sentence-transformers
+- sparse-encoder
+- sparse
+- splade
+- generated_from_trainer
+- dataset_size:100000
+- loss:SpladeLoss
+- loss:SparseMultipleNegativesRankingLoss
+- loss:FlopsLoss
+base_model: prajjwal1/bert-tiny
+widget:
+- text: 'The cause of simple liver cysts isn''t known, but they may be the result
+    of a malformation present at birth. Rarely, liver cysts may indicate a serious,
+    underlying condition such as: 1  Polycystic liver disease, an inherited disorder.  Echinococcus
+    infection, a parasitic infection.'
+- text: 'Foods that have roughly the same number of calories from fats, calories,
+    and protein will be found closer to the center of the pyramid. Read more about
+    the Caloric Ratio Pyramid. Caloric Ratio Pyramid for Apples, raw, with skin [Includes
+    USDA commodity food A343] Nutrition Facts & Calories. - In Internet Explorer 7
+    you will need to adjust the default Shrink To Fit setting. Note: Printing via
+    Mac Firefox is currently not supported. - In Internet Explorer 7 you will need
+    to adjust the default Shrink To Fit setting.'
+- text: average age virginity loss
+- text: verb (used with object), subsidized, subsidizing. 1. to furnish or aid with
+    a subsidy. 2. to purchase the assistance of by the payment of a subsidy. 3. to
+    secure the cooperation of by bribery; buy over. Also, especially British, subsidise.
+- text: benefits of health care act
+pipeline_tag: feature-extraction
+library_name: sentence-transformers
+metrics:
+- dot_accuracy@1
+- dot_accuracy@3
+- dot_accuracy@5
+- dot_accuracy@10
+- dot_precision@1
+- dot_precision@3
+- dot_precision@5
+- dot_precision@10
+- dot_recall@1
+- dot_recall@3
+- dot_recall@5
+- dot_recall@10
+- dot_ndcg@10
+- dot_mrr@10
+- dot_map@100
+- query_active_dims
+- query_sparsity_ratio
+- corpus_active_dims
+- corpus_sparsity_ratio
+model-index:
+- name: SPLADE-BERT-Tiny
+  results:
+  - task:
+      type: sparse-information-retrieval
+      name: Sparse Information Retrieval
+    dataset:
+      name: Unknown
+      type: unknown
+    metrics:
+    - type: dot_accuracy@1
+      value: 0.7258
+      name: Dot Accuracy@1
+    - type: dot_accuracy@3
+      value: 0.8584
+      name: Dot Accuracy@3
+    - type: dot_accuracy@5
+      value: 0.8986
+      name: Dot Accuracy@5
+    - type: dot_accuracy@10
+      value: 0.9372
+      name: Dot Accuracy@10
+    - type: dot_precision@1
+      value: 0.7258
+      name: Dot Precision@1
+    - type: dot_precision@3
+      value: 0.2861333333333333
+      name: Dot Precision@3
+    - type: dot_precision@5
+      value: 0.17972
+      name: Dot Precision@5
+    - type: dot_precision@10
+      value: 0.09371999999999998
+      name: Dot Precision@10
+    - type: dot_recall@1
+      value: 0.7258
+      name: Dot Recall@1
+    - type: dot_recall@3
+      value: 0.8584
+      name: Dot Recall@3
+    - type: dot_recall@5
+      value: 0.8986
+      name: Dot Recall@5
+    - type: dot_recall@10
+      value: 0.9372
+      name: Dot Recall@10
+    - type: dot_ndcg@10
+      value: 0.8335744020505143
+      name: Dot Ndcg@10
+    - type: dot_mrr@10
+      value: 0.8001331746031765
+      name: Dot Mrr@10
+    - type: dot_map@100
+      value: 0.8025817896431575
+      name: Dot Map@100
+    - type: query_active_dims
+      value: 34.04159927368164
+      name: Query Active Dims
+    - type: query_sparsity_ratio
+      value: 0.9988846864794678
+      name: Query Sparsity Ratio
+    - type: corpus_active_dims
+      value: 192.54740061035156
+      name: Corpus Active Dims
+    - type: corpus_sparsity_ratio
+      value: 0.9936915208501949
+      name: Corpus Sparsity Ratio
+---
+
+# SPLADE-BERT-Tiny
+
+This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [prajjwal1/bert-tiny](https://huggingface.co/prajjwal1/bert-tiny) using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space   and can be used for semantic search and sparse retrieval.
+## Model Details
+
+### Model Description
+- **Model Type:** SPLADE Sparse Encoder
+- **Base model:** [prajjwal1/bert-tiny](https://huggingface.co/prajjwal1/bert-tiny) <!-- at revision 6f75de8b60a9f8a2fdf7b69cbd86d9e64bcb3837 -->
+- **Maximum Sequence Length:** 512 tokens
+- **Output Dimensionality:** 30522 dimensions
+- **Similarity Function:** Dot Product
+<!-- - **Training Dataset:** Unknown -->
+- **Language:** en
+- **License:** mit
+
+### Model Sources
+
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder)
+
+### Full Model Architecture
+
+```
+SparseEncoder(
+  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
+  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
+)
+```
+
+## Usage
+
+### Direct Usage (Sentence Transformers)
+
+First install the Sentence Transformers library:
+
+```bash
+pip install -U sentence-transformers
+```
+
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SparseEncoder
+
+# Download from the 🤗 Hub
+model = SparseEncoder("rasyosef/SPLADE-BERT-Tiny")
+# Run inference
+queries = [
+    "benefits of health care act",
+]
+documents = [
+    'Every health insurance plan sold in the Marketplace will offer 10 essential health benefits. These essential health benefits include at least the following items and services: 1  Outpatient care—the kind you get without being admitted to a hospital.  Trips to the emergency room.',
+    'Essential Health Benefits. A set of 10 categories of services health insurance plans must cover under the Affordable Care Act. These include doctors’ services, inpatient and outpatient hospital care, prescription drug coverage, pregnancy and childbirth, mental health services, and more. Some plans cover more services.',
+    'Find Baker Hughes in Minot, ND 58701-5870 on Yellowbook. Get contact details or leave a review about this business.',
+]
+query_embeddings = model.encode_query(queries)
+document_embeddings = model.encode_document(documents)
+print(query_embeddings.shape, document_embeddings.shape)
+# [1, 30522] [3, 30522]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(query_embeddings, document_embeddings)
+print(similarities)
+# tensor([[13.6736, 18.2270,  0.0508]])
+```
+
+<!--
+### Direct Usage (Transformers)
+
+<details><summary>Click to see the direct usage in Transformers</summary>
+
+</details>
+-->
+
+<!--
+### Downstream Usage (Sentence Transformers)
+
+You can finetune this model on your own dataset.
+
+<details><summary>Click to expand</summary>
+
+</details>
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+## Evaluation
+
+### Metrics
+
+#### Sparse Information Retrieval
+
+* Evaluated with [<code>SparseInformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator)
+
+| Metric                | Value      |
+|:----------------------|:-----------|
+| dot_accuracy@1        | 0.7258     |
+| dot_accuracy@3        | 0.8584     |
+| dot_accuracy@5        | 0.8986     |
+| dot_accuracy@10       | 0.9372     |
+| dot_precision@1       | 0.7258     |
+| dot_precision@3       | 0.2861     |
+| dot_precision@5       | 0.1797     |
+| dot_precision@10      | 0.0937     |
+| dot_recall@1          | 0.7258     |
+| dot_recall@3          | 0.8584     |
+| dot_recall@5          | 0.8986     |
+| dot_recall@10         | 0.9372     |
+| **dot_ndcg@10**       | **0.8336** |
+| dot_mrr@10            | 0.8001     |
+| dot_map@100           | 0.8026     |
+| query_active_dims     | 34.0416    |
+| query_sparsity_ratio  | 0.9989     |
+| corpus_active_dims    | 192.5474   |
+| corpus_sparsity_ratio | 0.9937     |
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Dataset
+
+#### Unnamed Dataset
+
+* Size: 100,000 training samples
+* Columns: <code>query</code>, <code>positive</code>, and <code>negative</code>
+* Approximate statistics based on the first 1000 samples:
+  |         | query                                                                            | positive                                                                            | negative                                                                            |
+  |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
+  | type    | string                                                                           | string                                                                              | string                                                                              |
+  | details | <ul><li>min: 4 tokens</li><li>mean: 8.07 tokens</li><li>max: 21 tokens</li></ul> | <ul><li>min: 21 tokens</li><li>mean: 81.21 tokens</li><li>max: 236 tokens</li></ul> | <ul><li>min: 20 tokens</li><li>mean: 79.54 tokens</li><li>max: 187 tokens</li></ul> |
+* Samples:
+  | query                                               | positive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | negative                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+  |:----------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+  | <code>can a spouse make you move out</code>         | <code>If you and your spouse purchased a home together or the house was bought during the course of the marriage, you likely cannot make a spouse move out during divorce.However, if your spouse is engaging in abusive behavior towards you or towards your children, then you could seek a protective order or an injunction.his would be the case if you purchased the home before marriage; if the home is in your name only and if your spouse has made no contributions to the home during the time you were husband and wife.</code>               | <code>Many courts will not award a final restraining order unless there’s been actual physical violence. If you get a temporary restraining order because your spouse threatened violence, it may not be enough to keep him out of the home permanently, pending your divorce.tep 1. Talk it out. Your spouse might not want to be in the house with you any more than you want to live there with him, but he may have some viable concerns about moving out. Try to come up with a plan to address those concerns.</code> |
+  | <code>can i get pregnant right before period</code> | <code>simran47. It is difficult to get pregnant right before your period, since ovulation takes place around 2 weeks before the next period. This is true among women who have regular menstrual cycles and ovulation can be predicted to occur around mid-cycle. However, some women have irregular cycles. If your ovulation takes place later than expected (mid-cycle), say on the 24th day of the cycle instead of around 14th day, and fertilization and implantation take place, then pregnancy can occur right before your expected period.</code> | <code>The question about whether you can get pregnant on your period or not is probably the most common of the three situations noted above. The answer is yes and no. The probability is that you would not get pregnant having sex during your period.</code>                                                                                                                                                                                                                                                             |
+  | <code>average training mileage rates</code>         | <code>Beginning on Jan. 1, 2014, the standard mileage rates for the use of a car (also vans, pickups or panel trucks) will be: 1  56 cents per mile for business miles driven.  23.5 cents per mile driven for medical or moving purposes.</code>                                                                                                                                                                                                                                                                                                          | <code>Schneider Increases Mileage Rates. Trucker to Trucker is the major online resource for sourcing trucks for sale and for selling all trucking equipment, including rigs and semi trailers. Schneider National is a name known to all truckers and operators and has been in business for over three-quarters of a century.</code>                                                                                                                                                                                      |
+* Loss: [<code>SpladeLoss</code>](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters:
+  ```json
+  {
+      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
+      "document_regularizer_weight": 0.001,
+      "query_regularizer_weight": 0.002
+  }
+  ```
+
+### Training Hyperparameters
+#### Non-Default Hyperparameters
+
+- `eval_strategy`: epoch
+- `per_device_train_batch_size`: 32
+- `per_device_eval_batch_size`: 32
+- `learning_rate`: 4e-05
+- `num_train_epochs`: 4
+- `lr_scheduler_type`: cosine
+- `warmup_ratio`: 0.025
+- `fp16`: True
+- `optim`: adamw_torch_fused
+- `batch_sampler`: no_duplicates
+
+#### All Hyperparameters
+<details><summary>Click to expand</summary>
+
+- `overwrite_output_dir`: False
+- `do_predict`: False
+- `eval_strategy`: epoch
+- `prediction_loss_only`: True
+- `per_device_train_batch_size`: 32
+- `per_device_eval_batch_size`: 32
+- `per_gpu_train_batch_size`: None
+- `per_gpu_eval_batch_size`: None
+- `gradient_accumulation_steps`: 1
+- `eval_accumulation_steps`: None
+- `torch_empty_cache_steps`: None
+- `learning_rate`: 4e-05
+- `weight_decay`: 0.0
+- `adam_beta1`: 0.9
+- `adam_beta2`: 0.999
+- `adam_epsilon`: 1e-08
+- `max_grad_norm`: 1.0
+- `num_train_epochs`: 4
+- `max_steps`: -1
+- `lr_scheduler_type`: cosine
+- `lr_scheduler_kwargs`: {}
+- `warmup_ratio`: 0.025
+- `warmup_steps`: 0
+- `log_level`: passive
+- `log_level_replica`: warning
+- `log_on_each_node`: True
+- `logging_nan_inf_filter`: True
+- `save_safetensors`: True
+- `save_on_each_node`: False
+- `save_only_model`: False
+- `restore_callback_states_from_checkpoint`: False
+- `no_cuda`: False
+- `use_cpu`: False
+- `use_mps_device`: False
+- `seed`: 42
+- `data_seed`: None
+- `jit_mode_eval`: False
+- `use_ipex`: False
+- `bf16`: False
+- `fp16`: True
+- `fp16_opt_level`: O1
+- `half_precision_backend`: auto
+- `bf16_full_eval`: False
+- `fp16_full_eval`: False
+- `tf32`: None
+- `local_rank`: 0
+- `ddp_backend`: None
+- `tpu_num_cores`: None
+- `tpu_metrics_debug`: False
+- `debug`: []
+- `dataloader_drop_last`: False
+- `dataloader_num_workers`: 0
+- `dataloader_prefetch_factor`: None
+- `past_index`: -1
+- `disable_tqdm`: False
+- `remove_unused_columns`: True
+- `label_names`: None
+- `load_best_model_at_end`: False
+- `ignore_data_skip`: False
+- `fsdp`: []
+- `fsdp_min_num_params`: 0
+- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
+- `fsdp_transformer_layer_cls_to_wrap`: None
+- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
+- `deepspeed`: None
+- `label_smoothing_factor`: 0.0
+- `optim`: adamw_torch_fused
+- `optim_args`: None
+- `adafactor`: False
+- `group_by_length`: False
+- `length_column_name`: length
+- `ddp_find_unused_parameters`: None
+- `ddp_bucket_cap_mb`: None
+- `ddp_broadcast_buffers`: False
+- `dataloader_pin_memory`: True
+- `dataloader_persistent_workers`: False
+- `skip_memory_metrics`: True
+- `use_legacy_prediction_loop`: False
+- `push_to_hub`: False
+- `resume_from_checkpoint`: None
+- `hub_model_id`: None
+- `hub_strategy`: every_save
+- `hub_private_repo`: None
+- `hub_always_push`: False
+- `hub_revision`: None
+- `gradient_checkpointing`: False
+- `gradient_checkpointing_kwargs`: None
+- `include_inputs_for_metrics`: False
+- `include_for_metrics`: []
+- `eval_do_concat_batches`: True
+- `fp16_backend`: auto
+- `push_to_hub_model_id`: None
+- `push_to_hub_organization`: None
+- `mp_parameters`: 
+- `auto_find_batch_size`: False
+- `full_determinism`: False
+- `torchdynamo`: None
+- `ray_scope`: last
+- `ddp_timeout`: 1800
+- `torch_compile`: False
+- `torch_compile_backend`: None
+- `torch_compile_mode`: None
+- `include_tokens_per_second`: False
+- `include_num_input_tokens_seen`: False
+- `neftune_noise_alpha`: None
+- `optim_target_modules`: None
+- `batch_eval_metrics`: False
+- `eval_on_start`: False
+- `use_liger_kernel`: False
+- `liger_kernel_config`: None
+- `eval_use_gather_object`: False
+- `average_tokens_across_devices`: False
+- `prompts`: None
+- `batch_sampler`: no_duplicates
+- `multi_dataset_batch_sampler`: proportional
+- `router_mapping`: {}
+- `learning_rate_mapping`: {}
+
+</details>
+
+### Training Logs
+| Epoch | Step  | Training Loss | dot_ndcg@10 |
+|:-----:|:-----:|:-------------:|:-----------:|
+| 1.0   | 3125  | 22.6481       | 0.7960      |
+| 2.0   | 6250  | 0.6743        | 0.8296      |
+| 3.0   | 9375  | 0.5636        | 0.8333      |
+| 4.0   | 12500 | 0.5257        | 0.8336      |
+| -1    | -1    | -             | 0.8336      |
+
+
+### Framework Versions
+- Python: 3.11.13
+- Sentence Transformers: 5.0.0
+- Transformers: 4.53.0
+- PyTorch: 2.6.0+cu124
+- Accelerate: 1.8.1
+- Datasets: 3.6.0
+- Tokenizers: 0.21.2
+
+## Citation
+
+### BibTeX
+
+#### Sentence Transformers
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
+    author = "Reimers, Nils and Gurevych, Iryna",
+    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
+    month = "11",
+    year = "2019",
+    publisher = "Association for Computational Linguistics",
+    url = "https://arxiv.org/abs/1908.10084",
+}
+```
+
+#### SpladeLoss
+```bibtex
+@misc{formal2022distillationhardnegativesampling,
+      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
+      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
+      year={2022},
+      eprint={2205.04733},
+      archivePrefix={arXiv},
+      primaryClass={cs.IR},
+      url={https://arxiv.org/abs/2205.04733},
+}
+```
+
+#### SparseMultipleNegativesRankingLoss
+```bibtex
+@misc{henderson2017efficient,
+    title={Efficient Natural Language Response Suggestion for Smart Reply},
+    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
+    year={2017},
+    eprint={1705.00652},
+    archivePrefix={arXiv},
+    primaryClass={cs.CL}
+}
+```
+
+#### FlopsLoss
+```bibtex
+@article{paria2020minimizing,
+    title={Minimizing flops to learn efficient sparse representations},
+    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
+    journal={arXiv preprint arXiv:2004.05665},
+    year={2020}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+--> 

config.json

@@ -0,0 +1,24 @@
+{
+  "architectures": [
+    "BertForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 128,
+  "initializer_range": 0.02,
+  "intermediate_size": 512,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 2,
+  "num_hidden_layers": 2,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.53.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SparseEncoder",
+  "__version__": {
+    "sentence_transformers": "5.0.0",
+    "transformers": "4.53.0",
+    "pytorch": "2.6.0+cu124"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "dot"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90fd50282273fc3ba0af07f264845f68fc9d5e0c23813961674d30b94db33f12
+size 17671560

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.sparse_encoder.models.MLMTransformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_SpladePooling",
+    "type": "sentence_transformers.sparse_encoder.models.SpladePooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 512,
+    "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}