config

README.md

@@ -0,0 +1,39 @@
+
+# Mana Tokenizer
+
+The Mana Tokenizer is a custom-trained SentencePiece tokenizer for Persian text, trained on a combination of the Persian Wikipedia and Ganjoor datasets. The tokenizer uses the Unigram model type, optimized for handling the unique characteristics of Persian text.
+
+## Special Tokens
+
+- **UNK Token:** `<unk>`
+- **BOS Token:** `<s>`
+- **EOS Token:** `</s>`
+- **PAD Token:** `<pad>`
+
+## Usage
+
+You can load this tokenizer using the Hugging Face `transformers` library as follows:
+
+```python
+from transformers import PreTrainedTokenizerFast
+
+tokenizer = PreTrainedTokenizerFast.from_pretrained("your-username/mana_tokenizer")
+
+text = "این یک تست است."
+encoded = tokenizer(text)
+print(f"Encoded: {encoded}")
+
+decoded = tokenizer.decode(encoded['input_ids'])
+print(f"Decoded: {decoded}")
+
+
+
+Statistics
+
+    Vocabulary Size: 199,997
+    Character Coverage: 99.9%
+    Total Number of Text Samples: 1,022,675
+
+License
+
+This tokenizer is licensed under the MIT License. 

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+    "unk_token": "<unk>",
+    "bos_token": "<s>",
+    "eos_token": "</s>",
+    "pad_token": "<pad>"
+}

tokenizer_config.json

@@ -0,0 +1,9 @@
+{
+    "model_type": "unigram",
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "unk_token_id": 0,
+    "pad_token_id": 3,
+    "do_lower_case": false,
+    "max_length": 512
+}