File size: 416 Bytes
6404abb e5c4555 6404abb e5c4555 6404abb e5c4555 6404abb e5c4555 6404abb e5c4555 6404abb e5c4555 6404abb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | # RSLM Kumru-based Reasoning Tokenizer
Base tokenizer: `vngrs-ai/Kumru-2B`
This tokenizer preserves core IDs:
- `<|pad|>` = 0
- `<|unknown|>` = 1
- `<|begin_of_text|>` = 2
- `<|end_of_text|>` = 3
- `<|system_instruction|>` = 4
- `<|system_instruction_end|>` = 5
Reasoning tokens:
- `<think>` = 50176
- `</think>` = 50177
System prompt is intended to appear once at the beginning of each rendered conversation.
|