TEMPLE2 / tokenizer /config.json
empero-ai's picture
Upload folder using huggingface_hub
b15c56d verified
{
"vocab_size": 8192,
"special_tokens": {
"pad_token": "<|pad|>",
"bos_token": "<|bos|>",
"eos_token": "<|eos|>",
"unk_token": "<|unk|>"
},
"pad_id": 0,
"bos_id": 1,
"eos_id": 2,
"unk_id": 3
}