msqark-tokenizer / special_tokens_map.json
ThingsAI's picture
BPE tokenizer 32768 vocab with thinking/tool tokens
f862712 verified
Raw
History Blame Contribute Delete
465 Bytes
{
"bos_token": "<|im_start|>",
"eos_token": "<|endoftext|>",
"pad_token": "<|pad|>",
"additional_special_tokens": [
"<|im_end|>",
"<|thinking|>",
"<|/thinking|>",
"<|tool_calls|>",
"<|/tool_calls|>",
"<|tool_response|>",
"<|/tool_response|>",
"<|code|>",
"<|/code|>",
"<|analysis|>",
"<|/analysis|>",
"<|result|>",
"<|/result|>",
"<|tool|>",
"<|system|>",
"<|user|>",
"<|assistant|>"
]
}