succinct-router / tokenizer_config.json
pmarquees's picture
Upload 11 files
338bb9c verified
Raw
History Blame Contribute Delete
183 Bytes
{
"vocab_size": 8192,
"model_vocab_size": 8192,
"special_tokens": {
"<PAD>": 0,
"<UNK>": 1,
"<BOS>": 2,
"<EOS>": 3,
"<ROUTE>": 4,
"<HEAD_TAIL>": 5
}
}