ViuAI-500M / tokenizer /meta.json
ViuAI's picture
Upload tokenizer/meta.json with huggingface_hub
bd13dc1 verified
Raw
History Blame Contribute Delete
166 Bytes
{
"original_vocab_size": 64000,
"new_vocab_size": 64003,
"special_tokens": {
"<|user|>": 64000,
"<|assistant|>": 64001,
"<|endofturn|>": 64002
}
}