Add RSLM tokenizer trained on AkademikDerlem/makaleler

README.md

@@ -1,49 +1,41 @@
 ---
-library_name: transformers
-tags:
-- causal-lm
-- turkish
-- rslm
-- mqa
-- long-context
-- custom-code
-license: apache-2.0
+library_name: tokenizers
+language:
+- tr
+license: cc-by-sa-4.0
 ---
 
-# RSLM-1B-Speed
+# RSLM Tokenizer
 
-Speed-first 990M civarı decoder-only SLM mimarisi.
+Byte-Level BPE tokenizer trained for RSLM.
 
-## Mimari
+## Training source
 
-- hidden_size: 2048
-- layers: 24
-- Q heads: 16
-- KV heads: 1 / MQA
-- head_dim: 128
-- intermediate_size: 4352
-- vocab_size: 65536
-- context target: 262144
-- original training context target: 8192
-- block: parallel attention + MLP
-- norm: pre-RMSNorm
-- activation: SwiGLU
-- local layers: window 4096
-- global layers 0-indexed: 5, 11, 17, 23
+- Dataset: `turkish-nlp-suite/AkademikDerlem`
+- Subset/config: `makaleler`
+- Split: `train`
+- Column: `text`
 
-## Notlar
+## Settings
 
-Bu repo şu an tokenizer içermeyebilir. Tokenizer sonraki aşamada eklenecek.
+- Vocab size: `65536`
+- Model max length: `262144`
+- Target estimated tokens: `500,000,000`
+- Seen chars: `2,000,013,236`
+- Estimated tokens seen: `500,003,309`
 
-Bu checkpoint random init olabilir. Eğitimli model değildir.
+## Special tokens
 
-## Loading
-
-```python
-from transformers import AutoModelForCausalLM, AutoConfig
-
-config = AutoConfig.from_pretrained("Efe2898/new-model", trust_remote_code=True)
-model = AutoModelForCausalLM.from_pretrained("Efe2898/new-model", trust_remote_code=True)
+```text
+<|pad|>
+<|bos|>
+<|eos|>
+<|unk|>
+<|system|>
+<|user|>
+<|assistant|>
+<|answer|>
+<|end|>
+<think>
+</think>
 ```
-
-> Kaggle ortamındaki transformers/huggingface_hub uyumsuzluğu yüzünden bu repo, notebook içinde transformers import etmeden oluşturulmuştur.

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>",
+  "pad_token": "<|pad|>",
+  "additional_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|answer|>",
+    "<|end|>",
+    "<think>",
+    "</think>"
+  ]
+}

tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "model_max_length": 262144,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "clean_up_tokenization_spaces": false,
+  "padding_side": "right",
+  "truncation_side": "right",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|eos|>",
+  "unk_token": "<|unk|>",
+  "pad_token": "<|pad|>",
+  "additional_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|answer|>",
+    "<|end|>",
+    "<think>",
+    "</think>"
+  ],
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}<|system|>\n{{ message['content'] }}<|end|>\n{% elif message['role'] == 'user' %}<|user|>\n{{ message['content'] }}<|end|>\n{% elif message['role'] == 'assistant' %}<|assistant|>\n{{ message['content'] }}<|end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>\n{% endif %}"
+}

tokenizer_stats.json

@@ -0,0 +1,18 @@
+{
+  "dataset_id": "turkish-nlp-suite/AkademikDerlem",
+  "config_name": "makaleler",
+  "split": "train",
+  "text_column": "text",
+  "target_est_tokens": 500000000,
+  "chars_per_token_est": 4.0,
+  "target_chars": 2000000000,
+  "seen_rows": 98246,
+  "used_rows": 98238,
+  "seen_chars": 2000013236,
+  "skipped_rows": 8,
+  "started_at": "2026-05-07T21:31:05.276537Z",
+  "ended_at": "2026-05-07T21:40:07.755856Z",
+  "seconds": 610.76,
+  "estimated_tokens_seen": 500003309,
+  "final_vocab_size": 65536
+}