Update fine-tuned tramy-encoder

1_Pooling/config.json

@@ -1,10 +1,5 @@
 {
-    "word_embedding_dimension": 384,
-    "pooling_mode_cls_token": false,
-    "pooling_mode_mean_tokens": true,
-    "pooling_mode_max_tokens": false,
-    "pooling_mode_mean_sqrt_len_tokens": false,
-    "pooling_mode_weightedmean_tokens": false,
-    "pooling_mode_lasttoken": false,
+    "embedding_dimension": 384,
+    "pooling_mode": "mean",
     "include_prompt": true
 }

README.md

@@ -1,390 +1,57 @@
 ---
+language:
+- vi
+- en
+license: apache-2.0
 tags:
 - sentence-transformers
-- sentence-similarity
 - feature-extraction
-- dense
-- generated_from_trainer
-- dataset_size:1616
-- loss:CosineSimilarityLoss
-base_model: sentence-transformers/all-MiniLM-L6-v2
-widget:
-- source_sentence: Phường Bến Thành thuộc Thành phố nào?
-  sentences:
-  - Phường Kỳ Sơn thuộc Tỉnh Phú Thọ
-  - Phường Phú Thọ thuộc Tỉnh Phú Thọ
-  - Xã Bà Điểm thuộc Thành phố Hồ Chí Minh
-- source_sentence: Phường Bình Phú thuộc Thành phố nào?
-  sentences:
-  - Xã Xuân Lộc thuộc Tỉnh Đồng Nai
-  - Phường An Hội thuộc Tỉnh nào?
-  - 'Phường Cư Bao được sáp nhập từ: Phường Bình Tân (thị xã Buôn Hồ), Xã Bình Thuận
-    (thị xã Buôn Hồ) và Xã Cư Bao (thị xã Buôn Hồ) vào ngày 01/07/2025'
-- source_sentence: Xã Đại Đồng thuộc Tỉnh nào?
-  sentences:
-  - Phường Tam Quan thuộc Tỉnh nào?
-  - Phường Tân Thành thuộc Tỉnh Cà Mau
-  - Xã Đại Đồng thuộc Tỉnh Nghệ An
-- source_sentence: Xã Thái Mỹ thuộc Thành phố nào?
-  sentences:
-  - Xã Thái Mỹ thuộc Thành phố Hồ Chí Minh
-  - Phường Kiến Hưng được sáp nhập từ các Phường nào?
-  - Phường Hà Tiên được sáp nhập từ các Phường nào?
-- source_sentence: Phường Nghĩa Lộ được sáp nhập từ các Phường nào?
-  sentences:
-  - 'Xã Ngô Mây được sáp nhập từ: Xã Cát Hưng (huyện Phù Cát), Xã Cát Thắng (huyện
-    Phù Cát) và Xã Cát Chánh (huyện Phù Cát) vào ngày 01/07/2025'
-  - 'Xã Ngô Mây được sáp nhập từ: Xã Cát Hưng (huyện Phù Cát), Xã Cát Thắng (huyện
-    Phù Cát) và Xã Cát Chánh (huyện Phù Cát) vào ngày 01/07/2025'
-  - Phường Cao Lãnh được sáp nhập từ các Phường nào?
-pipeline_tag: sentence-similarity
-library_name: sentence-transformers
-metrics:
-- pearson_cosine
-- spearman_cosine
-model-index:
-- name: SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
-  results:
-  - task:
-      type: semantic-similarity
-      name: Semantic Similarity
-    dataset:
-      name: qa similarity eval
-      type: qa-similarity-eval
-    metrics:
-    - type: pearson_cosine
-      value: 0.898329291145673
-      name: Pearson Cosine
-    - type: spearman_cosine
-      value: 0.8699420206459687
-      name: Spearman Cosine
+- sentence-similarity
+- qdrant
+- vietnamese
+- tourist-notebook
+pipeline_tag: feature-extraction
 ---
 
-# SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
+# tramy-encoder
 
-This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2). It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
+This is a fine-tuned **alphaedge-ai/multilingual-e5-small-vie-32768** embedding model, specially optimized for **Vietnamese Tourist & Q&A Assistant Retrieval** with vector databases such as **Qdrant**.
 
 ## Model Details
+- **Base model**: `alphaedge-ai/multilingual-e5-small-vie-32768` (Vietnamese-trimmed vocabulary 32,768 tokens, ~137MB)
+- **Language**: Vietnamese (vi), English (en)
+- **Embedding dimension**: 384
+- **Primary use case**: Asymmetric retrieval (User search queries -> Stored Instructions/Documents/Passages)
 
-### Model Description
-- **Model Type:** Sentence Transformer
-- **Base model:** [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) <!-- at revision c9745ed1d9f207416be6d2e6f8de32d1f16199bf -->
-- **Maximum Sequence Length:** 256 tokens
-- **Output Dimensionality:** 384 dimensions
-- **Similarity Function:** Cosine Similarity
-<!-- - **Training Dataset:** Unknown -->
-<!-- - **Language:** Unknown -->
-<!-- - **License:** Unknown -->
-
-### Model Sources
-
-- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
-- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
-- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
+## ⚠️ Important Prefix Usage (E5 Architecture)
+Because this model uses the E5 architecture, you **must prepend prefixes**:
 
-### Full Model Architecture
-
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
-  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
-  (2): Normalize()
-)
-```
+- **When storing in Qdrant (Documents / Instructions / Answers)**: Prepend `passage: `
+  - Example: `passage: Thông tin tuyến xe buýt ở hồ chí minh`
+- **When searching in Qdrant (User queries)**: Prepend `query: `
+  - Example: `query: tìm tuyến xe buýt ở hồ chí minh`
 
-## Usage
+## Usage with sentence-transformers
 
-### Direct Usage (Sentence Transformers)
-
-First install the Sentence Transformers library:
-
-```bash
-pip install -U sentence-transformers
-```
-
-Then you can load this model and run inference.
 ```python
 from sentence_transformers import SentenceTransformer
+from sklearn.metrics.pairwise import cosine_similarity
 
-# Download from the 🤗 Hub
-model = SentenceTransformer("lmtri0312/tramy-encoder")
-# Run inference
-sentences = [
-    'Phường Nghĩa Lộ được sáp nhập từ các Phường nào?',
-    'Xã Ngô Mây được sáp nhập từ: Xã Cát Hưng (huyện Phù Cát), Xã Cát Thắng (huyện Phù Cát) và Xã Cát Chánh (huyện Phù Cát) vào ngày 01/07/2025',
-    'Xã Ngô Mây được sáp nhập từ: Xã Cát Hưng (huyện Phù Cát), Xã Cát Thắng (huyện Phù Cát) và Xã Cát Chánh (huyện Phù Cát) vào ngày 01/07/2025',
-]
-embeddings = model.encode(sentences)
-print(embeddings.shape)
-# [3, 384]
-
-# Get the similarity scores for the embeddings
-similarities = model.similarity(embeddings, embeddings)
-print(similarities)
-# tensor([[1.0000, 0.3627, 0.3627],
-#         [0.3627, 1.0000, 1.0000],
-#         [0.3627, 1.0000, 1.0000]])
-```
-
-<!--
-### Direct Usage (Transformers)
-
-<details><summary>Click to see the direct usage in Transformers</summary>
-
-</details>
--->
-
-<!--
-### Downstream Usage (Sentence Transformers)
-
-You can finetune this model on your own dataset.
-
-<details><summary>Click to expand</summary>
-
-</details>
--->
-
-<!--
-### Out-of-Scope Use
-
-*List how the model may foreseeably be misused and address what users ought not to do with the model.*
--->
-
-## Evaluation
-
-### Metrics
-
-#### Semantic Similarity
+# Load model (384 dimensions)
+model = SentenceTransformer('lmtri0312/tramy-encoder')
 
-* Dataset: `qa-similarity-eval`
-* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
-
-| Metric              | Value      |
-|:--------------------|:-----------|
-| pearson_cosine      | 0.8983     |
-| **spearman_cosine** | **0.8699** |
-
-<!--
-## Bias, Risks and Limitations
-
-*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
--->
-
-<!--
-### Recommendations
-
-*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
--->
-
-## Training Details
-
-### Training Dataset
-
-#### Unnamed Dataset
-
-* Size: 1,616 training samples
-* Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>label</code>
-* Approximate statistics based on the first 1000 samples:
-  |         | sentence_0                                                                        | sentence_1                                                                         | label                                                          |
-  |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------|
-  | type    | string                                                                            | string                                                                             | float                                                          |
-  | details | <ul><li>min: 5 tokens</li><li>mean: 19.64 tokens</li><li>max: 48 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 41.56 tokens</li><li>max: 224 tokens</li></ul> | <ul><li>min: 0.1</li><li>mean: 0.73</li><li>max: 1.0</li></ul> |
-* Samples:
-  | sentence_0                                                   | sentence_1                                                                                                                                                      | label            |
-  |:-------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------|
-  | <code>Phường Ayun Pa được sáp nhập từ các Phường nào?</code> | <code>Bão số 5 (Kajiki) diễn ra từ 00:00 24-08-2025 đến 23:59 27-08-2025, ở Thành phố Đà Nẵng</code>                                                            | <code>0.1</code> |
-  | <code>Xã Bà Điểm được sáp nhập từ các Xã nào?</code>         | <code>Xã Bà Điểm được sáp nhập từ: Xã Xuân Thới Thượng (huyện Hóc Môn), Xã Trung Chánh (huyện Hóc Môn) và Xã Bà Điểm (huyện Hóc Môn) vào ngày 01/07/2025</code> | <code>1.0</code> |
-  | <code>Phường An Biên được sáp nhập từ các Phường nào?</code> | <code>Xã Bình Mỹ được sáp nhập từ các Xã nào?</code>                                                                                                            | <code>0.7</code> |
-* Loss: [<code>CosineSimilarityLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosinesimilarityloss) with these parameters:
-  ```json
-  {
-      "loss_fct": "torch.nn.modules.loss.MSELoss"
-  }
-  ```
-
-### Training Hyperparameters
-#### Non-Default Hyperparameters
-
-- `eval_strategy`: steps
-- `num_train_epochs`: 4
-- `multi_dataset_batch_sampler`: round_robin
-
-#### All Hyperparameters
-<details><summary>Click to expand</summary>
-
-- `overwrite_output_dir`: False
-- `do_predict`: False
-- `eval_strategy`: steps
-- `prediction_loss_only`: True
-- `per_device_train_batch_size`: 8
-- `per_device_eval_batch_size`: 8
-- `per_gpu_train_batch_size`: None
-- `per_gpu_eval_batch_size`: None
-- `gradient_accumulation_steps`: 1
-- `eval_accumulation_steps`: None
-- `torch_empty_cache_steps`: None
-- `learning_rate`: 5e-05
-- `weight_decay`: 0.0
-- `adam_beta1`: 0.9
-- `adam_beta2`: 0.999
-- `adam_epsilon`: 1e-08
-- `max_grad_norm`: 1
-- `num_train_epochs`: 4
-- `max_steps`: -1
-- `lr_scheduler_type`: linear
-- `lr_scheduler_kwargs`: {}
-- `warmup_ratio`: 0.0
-- `warmup_steps`: 0
-- `log_level`: passive
-- `log_level_replica`: warning
-- `log_on_each_node`: True
-- `logging_nan_inf_filter`: True
-- `save_safetensors`: True
-- `save_on_each_node`: False
-- `save_only_model`: False
-- `restore_callback_states_from_checkpoint`: False
-- `no_cuda`: False
-- `use_cpu`: False
-- `use_mps_device`: False
-- `seed`: 42
-- `data_seed`: None
-- `jit_mode_eval`: False
-- `use_ipex`: False
-- `bf16`: False
-- `fp16`: False
-- `fp16_opt_level`: O1
-- `half_precision_backend`: auto
-- `bf16_full_eval`: False
-- `fp16_full_eval`: False
-- `tf32`: None
-- `local_rank`: 0
-- `ddp_backend`: None
-- `tpu_num_cores`: None
-- `tpu_metrics_debug`: False
-- `debug`: []
-- `dataloader_drop_last`: False
-- `dataloader_num_workers`: 0
-- `dataloader_prefetch_factor`: None
-- `past_index`: -1
-- `disable_tqdm`: False
-- `remove_unused_columns`: True
-- `label_names`: None
-- `load_best_model_at_end`: False
-- `ignore_data_skip`: False
-- `fsdp`: []
-- `fsdp_min_num_params`: 0
-- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
-- `fsdp_transformer_layer_cls_to_wrap`: None
-- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
-- `parallelism_config`: None
-- `deepspeed`: None
-- `label_smoothing_factor`: 0.0
-- `optim`: adamw_torch_fused
-- `optim_args`: None
-- `adafactor`: False
-- `group_by_length`: False
-- `length_column_name`: length
-- `ddp_find_unused_parameters`: None
-- `ddp_bucket_cap_mb`: None
-- `ddp_broadcast_buffers`: False
-- `dataloader_pin_memory`: True
-- `dataloader_persistent_workers`: False
-- `skip_memory_metrics`: True
-- `use_legacy_prediction_loop`: False
-- `push_to_hub`: False
-- `resume_from_checkpoint`: None
-- `hub_model_id`: None
-- `hub_strategy`: every_save
-- `hub_private_repo`: None
-- `hub_always_push`: False
-- `hub_revision`: None
-- `gradient_checkpointing`: False
-- `gradient_checkpointing_kwargs`: None
-- `include_inputs_for_metrics`: False
-- `include_for_metrics`: []
-- `eval_do_concat_batches`: True
-- `fp16_backend`: auto
-- `push_to_hub_model_id`: None
-- `push_to_hub_organization`: None
-- `mp_parameters`: 
-- `auto_find_batch_size`: False
-- `full_determinism`: False
-- `torchdynamo`: None
-- `ray_scope`: last
-- `ddp_timeout`: 1800
-- `torch_compile`: False
-- `torch_compile_backend`: None
-- `torch_compile_mode`: None
-- `include_tokens_per_second`: False
-- `include_num_input_tokens_seen`: False
-- `neftune_noise_alpha`: None
-- `optim_target_modules`: None
-- `batch_eval_metrics`: False
-- `eval_on_start`: False
-- `use_liger_kernel`: False
-- `liger_kernel_config`: None
-- `eval_use_gather_object`: False
-- `average_tokens_across_devices`: False
-- `prompts`: None
-- `batch_sampler`: batch_sampler
-- `multi_dataset_batch_sampler`: round_robin
-- `router_mapping`: {}
-- `learning_rate_mapping`: {}
-
-</details>
-
-### Training Logs
-| Epoch  | Step | Training Loss | qa-similarity-eval_spearman_cosine |
-|:------:|:----:|:-------------:|:----------------------------------:|
-| 0.4950 | 100  | -             | 0.8485                             |
-| 0.9901 | 200  | -             | 0.8517                             |
-| 1.0    | 202  | -             | 0.8517                             |
-| 1.4851 | 300  | -             | 0.8654                             |
-| 1.9802 | 400  | -             | 0.8693                             |
-| 2.0    | 404  | -             | 0.8691                             |
-| 2.4752 | 500  | 0.0429        | 0.8699                             |
-
-
-### Framework Versions
-- Python: 3.12.11
-- Sentence Transformers: 5.1.0
-- Transformers: 4.56.1
-- PyTorch: 2.8.0+cu126
-- Accelerate: 1.10.1
-- Datasets: 4.0.0
-- Tokenizers: 0.22.0
-
-## Citation
+# 1. Encoding documents to store in Qdrant
+docs = [
+    "Thông tin tuyến xe buýt ở hồ chí minh",
+    "Lịch trình du lịch Đà Lạt 3 ngày 2 đêm"
+]
+doc_embeddings = model.encode([f"passage: {d}" for d in docs], normalize_embeddings=True)
 
-### BibTeX
+# 2. Encoding user search query
+query = "tìm tuyến xe buýt ở hồ chí minh"
+query_embedding = model.encode([f"query: {query}"], normalize_embeddings=True)
 
-#### Sentence Transformers
-```bibtex
-@inproceedings{reimers-2019-sentence-bert,
-    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
-    author = "Reimers, Nils and Gurevych, Iryna",
-    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
-    month = "11",
-    year = "2019",
-    publisher = "Association for Computational Linguistics",
-    url = "https://arxiv.org/abs/1908.10084",
-}
+# 3. Calculate cosine similarity
+similarity = cosine_similarity(query_embedding, doc_embeddings)
+print("Similarity:", similarity)
 ```
-
-<!--
-## Glossary
-
-*Clearly define terms in order to be accessible across audiences.*
--->
-
-<!--
-## Model Card Authors
-
-*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
--->
-
-<!--
-## Model Card Contact
-
-*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
--->

config.json

@@ -1,25 +1,30 @@
 {
+  "add_cross_attention": false,
   "architectures": [
     "BertModel"
   ],
   "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
   "classifier_dropout": null,
   "dtype": "float32",
-  "gradient_checkpointing": false,
+  "eos_token_id": 2,
   "hidden_act": "gelu",
   "hidden_dropout_prob": 0.1,
   "hidden_size": 384,
   "initializer_range": 0.02,
   "intermediate_size": 1536,
+  "is_decoder": false,
   "layer_norm_eps": 1e-12,
   "max_position_embeddings": 512,
   "model_type": "bert",
   "num_attention_heads": 12,
-  "num_hidden_layers": 6,
-  "pad_token_id": 0,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
   "position_embedding_type": "absolute",
-  "transformers_version": "4.56.1",
+  "tie_word_embeddings": true,
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "transformers_version": "5.13.1",
   "type_vocab_size": 2,
-  "use_cache": true,
-  "vocab_size": 30522
+  "use_cache": false,
+  "vocab_size": 32768
 }

config_sentence_transformers.json

@@ -1,14 +1,14 @@
 {
   "__version__": {
-    "sentence_transformers": "5.1.0",
-    "transformers": "4.56.1",
-    "pytorch": "2.8.0+cu126"
+    "pytorch": "2.11.0+cu128",
+    "sentence_transformers": "5.6.0",
+    "transformers": "5.13.1"
   },
+  "default_prompt_name": null,
   "model_type": "SentenceTransformer",
   "prompts": {
-    "query": "",
-    "document": ""
+    "document": "",
+    "query": ""
   },
-  "default_prompt_name": null,
   "similarity_fn_name": "cosine"
 }

eval/similarity_evaluation_e5-vietnamese-eval_results.csv

@@ -0,0 +1,9 @@
+epoch,steps,cosine_pearson,cosine_spearman
+1.0,3324,0.9602423552105073,0.2795829289673402
+2.0,6648,0.9663303369405396,0.2795882717620926
+3.0,9972,0.9719438552530223,0.2795965265730658
+4.0,13296,0.9709707717460083,0.2795965268328552
+1.0,3324,0.9716069995001885,0.2795950674475349
+2.0,6648,0.9671237949811639,0.2795873006113885
+3.0,9972,0.9642597483991596,0.2795853599057664
+4.0,13296,0.9685522197371857,0.2795873018217958

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b018efe0ab6b14c4fdd4fea975796f9dfe6102fa658883961b4a0458030bce47
-size 90864192
+oid sha256:bca06e721075aaf44e05cb1830d686c2e2d4dd0b58079199aadadaa60feca4b6
+size 136912008

modules.json

@@ -3,18 +3,18 @@
     "idx": 0,
     "name": "0",
     "path": "",
-    "type": "sentence_transformers.models.Transformer"
+    "type": "sentence_transformers.base.modules.transformer.Transformer"
   },
   {
     "idx": 1,
     "name": "1",
     "path": "1_Pooling",
-    "type": "sentence_transformers.models.Pooling"
+    "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
   },
   {
     "idx": 2,
     "name": "2",
     "path": "2_Normalize",
-    "type": "sentence_transformers.models.Normalize"
+    "type": "sentence_transformers.sentence_transformer.modules.normalize.Normalize"
   }
 ]

sentence_bert_config.json

@@ -1,4 +1,10 @@
 {
-    "max_seq_length": 256,
-    "do_lower_case": false
+    "transformer_task": "feature-extraction",
+    "modality_config": {
+        "text": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state"
+        }
+    },
+    "module_output_name": "token_embeddings"
 }

tokenizer_config.json

@@ -1,65 +1,16 @@
 {
-  "added_tokens_decoder": {
-    "0": {
-      "content": "[PAD]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "100": {
-      "content": "[UNK]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "101": {
-      "content": "[CLS]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "102": {
-      "content": "[SEP]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "103": {
-      "content": "[MASK]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    }
-  },
-  "clean_up_tokenization_spaces": false,
-  "cls_token": "[CLS]",
-  "do_basic_tokenize": true,
-  "do_lower_case": true,
-  "extra_special_tokens": {},
-  "mask_token": "[MASK]",
-  "max_length": 128,
-  "model_max_length": 256,
-  "never_split": null,
-  "pad_to_multiple_of": null,
-  "pad_token": "[PAD]",
-  "pad_token_type_id": 0,
-  "padding_side": "right",
-  "sep_token": "[SEP]",
-  "stride": 0,
-  "strip_accents": null,
-  "tokenize_chinese_chars": true,
-  "tokenizer_class": "BertTokenizer",
-  "truncation_side": "right",
-  "truncation_strategy": "longest_first",
-  "unk_token": "[UNK]"
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
 }