Initial model upload - clean repository

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model_performance_2d.png filter=lfs diff=lfs merge=lfs -text
+post_train_retrieval_2d.png filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+    "word_embedding_dimension": 1024,
+    "pooling_mode_cls_token": false,
+    "pooling_mode_mean_tokens": true,
+    "pooling_mode_max_tokens": false,
+    "pooling_mode_mean_sqrt_len_tokens": false,
+    "pooling_mode_weightedmean_tokens": false,
+    "pooling_mode_lasttoken": false,
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,326 @@
+---
+language:
+- tr
+- en
+library_name: sentence-transformers
+license: apache-2.0
+tags:
+- sentence-transformers
+- sentence-similarity
+- feature-extraction
+- information-retrieval
+- dense-retrieval
+- turkish
+- legal
+- turkish-legal
+- mecellem
+- modernbert
+- TRUBA
+- MN5
+datasets:
+- newmindai/ms-marco-turkish-triplets
+- newmindai/EuroHPC-Legal
+base_model: newmindai/Mursit-Large
+metrics:
+- ndcg@10
+- mrr@10
+- map@100
+pipeline_tag: sentence-similarity
+---
+
+# Mursit-Large-TR-Retrieval
+
+[![GitHub](https://img.shields.io/badge/GitHub-NewMindAI-black?logo=github)](https://github.com/newmindai/mecellem-models) [![HuggingFace Space](https://img.shields.io/badge/HF%20Space-Mizan-blue?logo=huggingface)](https://huggingface.co/spaces/newmindai/Mizan) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+
+## Model Description
+
+Mursit-Large-TR-Retrieval is a large-scale Turkish embedding model pre-trained entirely from scratch on Turkish-dominant corpora and fine-tuned for retrieval tasks. The model is based on ModernBERT-large architecture (403M parameters) and optimized specifically for Turkish legal domain applications. This model achieves strong performance on Turkish retrieval benchmarks with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models.
+
+**Key Features:**
+- Pre-trained from scratch on approximately 112.7 billion tokens of Turkish-dominant corpus
+- Post-trained for embedding tasks using contrastive learning on MS MARCO-TR dataset
+- Achieves strong performance on Turkish legal retrieval benchmarks
+- Optimized for Turkish legal domain with custom tokenizer trained on legal documents
+
+**Model Type:** Embedding  
+**Parameters:** 403M  
+**Base Model:** newmindai/Mursit-Large  
+**Architecture:** ModernBERT-large  
+**Embedding Dimension:** 1,024  
+**Max Sequence Length:** 2,048 tokens
+
+### Architecture Details
+
+The model is based on ModernBERT-large architecture:
+
+- **Attention Mechanism:** Alternating local and global attention
+- **Normalization:** Pre-layer normalization with RMSNorm
+- **Activation:** GeGLU (Gated Linear Units with GELU) in MLP layers
+- **Position Embeddings:** Rotary positional embeddings (RoPE) with θ=20,000
+- **Context Length:** 2,048 tokens
+- **Layers:** 28 transformer layers
+- **Hidden Size:** 1,024
+- **FFN Size:** 2,624
+- **Attention Heads:** 16 heads with 64 dimensions each
+- **Window Size:** 128 (for sliding window attention in local layers)
+- **Vocabulary Size:** 59,008 tokens
+
+### Training Details
+
+**Pre-training:**
+- **Dataset:** Turkish-dominant corpus totaling approximately 112.7 billion tokens
+  - **Legal Sources:** 
+    - Court of Cassation (Yargıtay): 10.3M sequences, ~3.43B tokens
+    - Council of State (Danıştay): 151K sequences, ~0.11B tokens
+    - Academic theses (YÖKTEZ): 21.1M sequences, ~9.61B tokens (after DocsOCR processing)
+  - **General Turkish Sources:**
+    - FineWeb2: General Turkish web data
+    - CulturaX: Multilingual corpus (Turkish subset)
+    - Total general Turkish: 212M sequences, ~96.17B tokens
+  - **Data Processing:** SemHash-based semantic deduplication, FineWeb quality filtering, URL-based filtering, page-packing for YÖKTEZ documents
+- **Training Method:** Masked Language Modeling (MLM) with 15% masking probability
+- **Masking Strategy:** 80% [MASK], 10% random token, 10% unchanged (80-10-10 strategy)
+- **Framework:** MosaicML Composer with Decoupled StableAdamW optimizer
+- **Learning Rate:** 8×10⁻⁴ with warmup_stable_decay schedule
+- **Precision:** BF16 mixed precision
+- **Hardware Infrastructure:**
+  - **System:** MareNostrum 5 ACC partition at Barcelona Supercomputing Center (BSC)
+  - **Compute Nodes:** 32 nodes
+  - **GPUs:** 128× NVIDIA Hopper H100 64GB GPUs (4 GPUs per node)
+  - **Node Configuration:** Each node equipped with 4× H100 GPUs, 80 CPU cores, 512GB DDR5 memory
+  - **Interconnect:** 800 Gb/s InfiniBand for distributed training
+  - **GPU Interconnect:** NVLink for intra-node GPU communication (4 GPUs per node connected via NVLink)
+  - **Distributed Training:** Multi-node distributed training across 32 nodes with InfiniBand interconnect
+
+**Post-training for Embeddings:**
+- **Dataset:** MS MARCO-TR (920,106 triplets)
+- **Loss Function:** CachedGISTEmbedLoss with BGE-M3 guide model (568M parameters, 1024-dimensional embeddings)
+- **Training Framework:** Sentence Transformers
+- **Optimization:** Contrastive learning on Turkish passage ranking dataset
+- **Hardware:** 4× H100 GPUs (single node, NVLink interconnect)
+- **Optimizer:** AdamW (learning rate: 2×10⁻⁵, weight decay: 0.01)
+
+## Performance on MTEB-Turkish Benchmark
+
+The following visualization shows the model's performance compared to other Turkish language models:
+
+![Model Performance Comparison](model_performance_2d.png)
+
+*Model Performance Comparison: Legal Score vs. MTEB Score. Embedding models (green triangles) show superior performance compared to MLM models. Mursit-Large-TR-Retrieval achieves strong performance with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models.*
+
+This model was evaluated on the comprehensive MTEB-Turkish benchmark, which includes 17 tasks across 5 task types. The benchmark evaluates models on general Turkish NLP tasks as well as domain-specific legal retrieval tasks.
+
+### Comprehensive Benchmark Results
+
+The following table presents comprehensive evaluation results across all models evaluated on the MTEB-Turkish benchmark. *This model's results are highlighted in italics.*
+
+| Model | MTEB | Legal | Cls. | Clus. | Pair | Ret. | STS | Cont. | Reg. | Case | Params | Type |
+|-------|------|-------|------|-------|------|------|-----|-------|------|------|--------|------|
+| embeddinggemma-300m | **65.42** | 50.63 | **77.74** | **45.05** | **80.02** | **55.06** | 69.22 | 83.97 | **39.56** | 28.38 | 307M | Emb. |
+| bge-m3 | 62.87 | **51.16** | 75.35 | 35.86 | 78.88 | 54.42 | **69.83** | **86.08** | 38.09 | **29.3** | 567M | Emb. |
+| Mursit-Embed-Qwen3-1.7B-TR | 56.84 | 34.76 | 68.46 | 42.22 | 59.67 | 50.1 | 63.77 | 70.22 | 17.94 | 16.11 | 1.7B | CLM-E. |
+| *Mursit-Large-TR-Retrieval* | 56.87 | 46.56 | 67.72 | 41.15 | 59.78 | 51.69 | 64.01 | 81.78 | 32.67 | 25.24 | 403M | Emb. |
+| Mursit-Base-TR-Retrieval | 55.86 | 47.52 | 66.25 | 39.75 | 61.31 | 50.07 | 61.9 | 80.4 | 34.1 | 28.07 | 155M | Emb. |
+| Mursit-Embed-Qwen3-4B-TR | 53.65 | 37.0 | 67.29 | 36.68 | 58.36 | 51.12 | 54.77 | 69.25 | 24.21 | 17.56 | 4B | CLM-E. |
+|-------|------|-------|------|------|------|------|-----|-------|------|------|--------|------|
+| bert-base-turkish-uncased | 46.23 | 24.94 | 68.05 | 33.81 | 60.44 | 32.01 | 36.85 | 52.47 | 12.05 | 10.29 | 110M | MLM |
+| turkish-large-bert-cased | 45.3 | 19.12 | 67.43 | 34.24 | 60.11 | 28.68 | 36.04 | 47.57 | 5.93 | 3.85 | 337M | MLM |
+| bert-base-turkish-cased | 45.17 | 24.41 | 66.39 | 35.28 | 60.05 | 30.52 | 33.62 | 54.03 | 10.13 | 9.07 | 110M | MLM |
+| BERTurk-Legal | 42.02 | 32.63 | 60.61 | 26.24 | 59.51 | 25.8 | 37.94 | 61.4 | 15.51 | 20.99 | 184M | MLM |
+| Mursit-Large | 41.75 | 23.71 | 62.95 | 25.34 | 58.04 | 27.4 | 35.01 | 42.74 | 11.29 | 17.1 | 403M | MLM |
+| turkish-base-bert-uncased | 44.68 | 27.58 | 66.22 | 30.23 | 58.84 | 31.4 | 36.74 | 56.6 | 13.39 | 12.74 | 110M | MLM |
+| Mursit-Base | 40.23 | 17.93 | 59.78 | 25.48 | 58.65 | 20.82 | 36.45 | 36.0 | 7.4 | 10.4 | 155M | MLM |
+| mmBERT-base | 39.65 | 12.15 | 61.84 | 26.77 | 59.25 | 15.83 | 34.56 | 34.45 | 1.33 | 0.68 | 306M | MLM |
+| TabiBERT | 37.77 | 11.5 | 59.63 | 25.75 | 58.19 | 14.96 | 30.32 | 32.02 | 1.86 | 0.63 | 148M | MLM |
+| ModernBERT-base | 23.8 | 2.99 | 39.06 | 2.01 | 53.95 | 2.1 | 21.91 | 7.92 | 0.62 | 0.43 | 149M | MLM |
+| ModernBERT-large | 23.74 | 2.44 | 39.44 | 3.9 | 53.73 | 1.8 | 19.85 | 6.12 | 0.62 | 0.59 | 394M | MLM |
+
+**Column abbreviations:** MTEB = mean performance across task types; Legal = weighted average of Contracts, Regulation, Caselaw; Classification = accuracy on Turkish classification tasks; Clustering = V-measure on clustering tasks; Pair Classification = average precision on pair classification tasks like NLI; Retrieval = nDCG@10 on information retrieval tasks; Semantic Textual Similarity = Spearman correlation; Contracts = nDCG@10 on legal contract retrieval; Regulation = nDCG@10 on regulatory text retrieval; Caselaw = nDCG@10 on case law retrieval; Number of Parameters = number of model parameters; Model Type = model type (Embedding, CLM-Embedding, Masked Language Model). **Bold values** indicate the highest score in each column.
+
+**Key Findings:**
+- The model achieves strong performance with 56.87 MTEB Score and 46.56 Legal Score, ranking among the top Turkish embedding models
+- Strong performance on Contracts retrieval (81.78 nDCG@10) demonstrates effectiveness for legal document search
+- Post-training on MS MARCO-TR significantly improves retrieval capabilities compared to base MLM models
+
+### Post-Training Performance Analysis
+
+The following visualization shows the impact of post-training on retrieval performance:
+
+![Post-Training Retrieval Performance](post_train_retrieval_2d.png)
+
+*Post-Training Retrieval Performance Comparison. Post-trained models (Mursit-Base-TR-Retrieval and Mursit-Large-TR-Retrieval) show significant improvements in legal domain retrieval tasks compared to base MLM models.*
+
+## Reproducibility
+
+To reproduce the benchmark results and training procedures for this model, please refer to:
+
+- **Post-Training:** [github.com/newmindai/mecellem-models/training/post-training-retrieval](https://github.com/newmindai/mecellem-models/tree/main/training/post-training-retrieval) - Contains code and configurations for post-training retrieval models on MS MARCO-TR dataset.
+- **Embedding Benchmark Results:** [github.com/newmindai/mecellem-models/benchmark/embedding_model](https://github.com/newmindai/mecellem-models/tree/main/benchmark/embedding_model) - Contains code and evaluation configurations for reproducing MTEB-Turkish benchmark results.
+
+## Usage
+
+### Installation
+
+```bash
+pip install sentence-transformers
+```
+
+### Basic Usage
+
+```python
+from sentence_transformers import SentenceTransformer
+
+# Load model
+model = SentenceTransformer("newmindai/Mursit-Large-TR-Retrieval")
+
+# Encode sentences
+sentences = [
+    "Türk hukuk sistemi medeni hukuk geleneğine dayanır",
+    "Anayasa Türkiye Cumhuriyeti'nin temel hukuk belgesidir"
+]
+
+embeddings = model.encode(sentences)
+print(embeddings.shape)  # (2, 1024)
+```
+
+### Information Retrieval
+
+```python
+from sentence_transformers import SentenceTransformer, util
+
+model = SentenceTransformer("newmindai/Mursit-Large-TR-Retrieval")
+
+query = "Sözleşme feshi nasıl yapılır?"
+documents = [
+    "Sözleşmeler yazılı olarak feshedilebilir.",
+    "İş kanunu çalışma koşullarını düzenler."
+]
+
+query_embedding = model.encode(query, convert_to_tensor=True)
+doc_embeddings = model.encode(documents, convert_to_tensor=True)
+scores = util.cos_sim(query_embedding, doc_embeddings)[0]
+
+results = [(doc, score.item()) for doc, score in zip(documents, scores)]
+results.sort(key=lambda x: x[1], reverse=True)
+
+for doc, score in results:
+    print(f"Score: {score:.4f} - {doc}")
+```
+# ONNX Model Inference
+
+This script demonstrates how to use the ONNX model from Hugging Face for text embedding generation.
+
+## Exporting Model to ONNX
+
+To export the model to ONNX format, use the `optimum-cli` command:
+
+```bash
+optimum-cli export onnx \
+  -m newmindai/Mursit-Large-TR-Retrieval \
+  --task feature-extraction \
+  onnx/MursitLargeTRRetrieval
+```
+
+This will create the `model.onnx` file in the specified output directory.
+
+## Installation
+
+```bash
+pip install onnxruntime-gpu transformers huggingface_hub numpy
+```
+
+## Usage
+
+```python
+import onnxruntime as ort
+from transformers import AutoTokenizer
+from huggingface_hub import hf_hub_download
+import numpy as np
+
+model_id = "newmindai/Mursit-Large-TR-Retrieval"
+
+# Load tokenizer and download ONNX model from Hugging Face
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+onnx_path = hf_hub_download(repo_id=model_id, filename="model.onnx")
+
+# Use GPU if available, otherwise fallback to CPU
+providers = ["CUDAExecutionProvider", "CPUExecutionProvider"] if "CUDAExecutionProvider" in ort.get_available_providers() else ["CPUExecutionProvider"]
+
+sess = ort.InferenceSession(onnx_path, providers=providers)
+
+texts = ["This is a test"]
+inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="np")
+
+outputs = sess.run(None, {
+    "input_ids": inputs["input_ids"].astype(np.int64),
+    "attention_mask": inputs["attention_mask"].astype(np.int64),
+})
+
+embeddings = outputs[-1]  # sentence_embedding is usually the last output
+print(embeddings.shape)
+print(embeddings[:1])
+```
+
+## Features
+
+- **Automatic GPU/CPU selection**: Uses CUDA if available, otherwise falls back to CPU
+- **Hugging Face integration**: Downloads model files directly from Hugging Face Hub
+- **Simple API**: Easy-to-use interface for text embedding generation
+
+## Use Cases
+
+- Semantic search in Turkish legal documents
+- Legal document retrieval and ranking
+- Contract similarity and matching
+- Regulation compliance checking
+- Case law research and discovery
+- Question answering systems for legal domain
+
+## Reproducibility
+
+To reproduce the benchmark results and training procedures for this model, please refer to:
+
+- **Post-Training:** [github.com/newmindai/mecellem-models/training/post-training-retrieval](https://github.com/newmindai/mecellem-models/tree/main/training/post-training-retrieval) - Contains code and configurations for post-training retrieval models on MS MARCO-TR dataset.
+- **Embedding Benchmark Results:** [github.com/newmindai/mecellem-models/benchmark/embedding_model](https://github.com/newmindai/mecellem-models/tree/main/benchmark/embedding_model) - Contains code and evaluation configurations for reproducing MTEB-Turkish benchmark results.
+
+## Acknowledgments
+
+This work was supported by the EuroHPC Joint Undertaking through project etur46 with access to the MareNostrum 5 supercomputer, hosted by Barcelona Supercomputing Center (BSC), Spain. MareNostrum 5 is owned by EuroHPC JU and operated by BSC. We are grateful to the BSC support team for their assistance with job scheduling, environment configuration, and technical guidance throughout the project.
+
+The numerical calculations reported in this work were fully/partially performed at TÜBİTAK ULAKBİM, High Performance and Grid Computing Center (TRUBA resources). The authors gratefully acknowledge the know-how provided by the MINERVA Support for expert guidance and collaboration opportunities in HPC-AI integration.
+
+## References
+
+If you use this model, please cite our paper:
+
+```bibtex
+@article{mecellem2026,
+  title={Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain},
+  author={Uğur, Özgür and Göksu, Mahmut and Şavirdi, Esra and Çimen, Mahmut and Yılmaz, Musa and Demir, Alp Talha and Güllüce, Rumeysa and Çetin, İclal and Sağbaş, Ömer Can},
+  journal={Procedia Computer Science},
+  year={2026},
+  publisher={Elsevier}
+}
+```
+### Base Model References
+
+```bibtex
+@inproceedings{modernbert2025,
+  title={ModernBERT: A Modern Bidirectional Encoder Transformer},
+  author={Answer.AI and LightOn},
+  booktitle={Proceedings of the 2025 Conference on Language Models},
+  year={2025}
+}
+```
+```bibtex
+@misc{bge-m3,
+  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation}, 
+  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng},
+  year={2024},
+  eprint={2402.03216},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL}
+}
+```
+
+<!-- Updated: 2026-01-15 09:38:18 -->

config.json

@@ -0,0 +1,45 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "classifier_activation": "silu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 2,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 20000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 20000.0,
+  "max_position_embeddings": 2048,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "sep_token_id": 2,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "transformers_version": "4.57.0",
+  "vocab_size": 59008
+}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "model_type": "SentenceTransformer",
+  "__version__": {
+    "sentence_transformers": "5.1.1",
+    "transformers": "4.57.0",
+    "pytorch": "2.8.0+cu128"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4b33709916375a1e3fe90463312b8da9d2fd54100fb7b9959e28018866361a20
+size 1615375340

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4188d15c1c6015778bbdea4fba35189125f46e4271dc5826d2d49f01633cbeea
+size 1614533136

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+    "max_seq_length": 2048,
+    "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 2048,
+  "pad_token": "[PAD]",
+  "sep_token": "</s>",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}